SoftTeacher
SoftTeacher copied to clipboard
Published
20 hours ago •
microsoft
microsoft
KeyError: 'SemiBalanceSampler is not in the sampler registry'
I try to run in Google Colab with 1 gpu but get error. My command for run:
python train.py /content/SoftTeacher/configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py \
--launcher none --cfg-options percent=10 fond=1 --gpus 1
Traceback:
2021-11-02 09:31:04,029 - mmdet.ssod - INFO - [<StreamHandler <stderr> (INFO)>, <FileHandler /content/SoftTeacher/tools/work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k/20211102_093103.log (INFO)>]
2021-11-02 09:31:04,029 - mmdet.ssod - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.7.12 (default, Sep 10 2021, 00:21:48) [GCC 7.5.0]
CUDA available: True
GPU 0: Tesla T4
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.1.TC455_06.29190527_0
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.9.0+cu111
PyTorch compiling details: PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 11.1
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
- CuDNN 8.0.5
- Magma 2.5.2
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
TorchVision: 0.10.0+cu111
OpenCV: 4.1.2
MMCV: 1.3.16
MMCV Compiler: GCC 7.5
MMCV CUDA Compiler: 11.1
MMDetection: 2.18.0+2f5b0df
------------------------------------------------------------
2021-11-02 09:31:06,848 - mmdet.ssod - INFO - Distributed training: False
2021-11-02 09:31:09,585 - mmdet.ssod - INFO - Config:
model = dict(
type='SoftTeacher',
model=dict(
type='FasterRCNN',
backbone=dict(
type='ResNet',
depth=50,
num_stages=4,
out_indices=(0, 1, 2, 3),
frozen_stages=1,
norm_cfg=dict(type='BN', requires_grad=False),
norm_eval=True,
style='caffe',
init_cfg=dict(
type='Pretrained',
checkpoint='open-mmlab://detectron2/resnet50_caffe')),
neck=dict(
type='FPN',
in_channels=[256, 512, 1024, 2048],
out_channels=256,
num_outs=5),
rpn_head=dict(
type='RPNHead',
in_channels=256,
feat_channels=256,
anchor_generator=dict(
type='AnchorGenerator',
scales=[8],
ratios=[0.5, 1.0, 2.0],
strides=[4, 8, 16, 32, 64]),
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[1.0, 1.0, 1.0, 1.0]),
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0)),
roi_head=dict(
type='StandardRoIHead',
bbox_roi_extractor=dict(
type='SingleRoIExtractor',
roi_layer=dict(
type='RoIAlign', output_size=7, sampling_ratio=0),
out_channels=256,
featmap_strides=[4, 8, 16, 32]),
bbox_head=dict(
type='Shared2FCBBoxHead',
in_channels=256,
fc_out_channels=1024,
roi_feat_size=7,
num_classes=80,
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[0.1, 0.1, 0.2, 0.2]),
reg_class_agnostic=False,
loss_cls=dict(
type='CrossEntropyLoss',
use_sigmoid=False,
loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0))),
train_cfg=dict(
rpn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.7,
neg_iou_thr=0.3,
min_pos_iou=0.3,
match_low_quality=True,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=256,
pos_fraction=0.5,
neg_pos_ub=-1,
add_gt_as_proposals=False),
allowed_border=-1,
pos_weight=-1,
debug=False),
rpn_proposal=dict(
nms_pre=2000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.5,
neg_iou_thr=0.5,
min_pos_iou=0.5,
match_low_quality=False,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=512,
pos_fraction=0.25,
neg_pos_ub=-1,
add_gt_as_proposals=True),
pos_weight=-1,
debug=False)),
test_cfg=dict(
rpn=dict(
nms_pre=1000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
score_thr=0.05,
nms=dict(type='nms', iou_threshold=0.5),
max_per_img=100))),
train_cfg=dict(
use_teacher_proposal=False,
pseudo_label_initial_score_thr=0.5,
rpn_pseudo_threshold=0.9,
cls_pseudo_threshold=0.9,
reg_pseudo_threshold=0.02,
jitter_times=10,
jitter_scale=0.06,
min_pseduo_box_size=0,
unsup_weight=2.0),
test_cfg=dict(inference_on='student'))
dataset_type = 'CocoDataset'
data_root = '/content/data/'
img_norm_cfg = dict(
mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False)
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5),
dict(
type='OneOf',
transforms=[
dict(type='Identity'),
dict(type='AutoContrast'),
dict(type='RandEqualize'),
dict(type='RandSolarize'),
dict(type='RandColor'),
dict(type='RandContrast'),
dict(type='RandBrightness'),
dict(type='RandSharpness'),
dict(type='RandPosterize')
])
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='sup'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg',
'pad_shape', 'scale_factor', 'tag'))
]
test_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]
data = dict(
samples_per_gpu=1,
workers_per_gpu=2,
train=dict(
type='SemiDataset',
sup=dict(
type='CocoDataset',
ann_file='/content/data/labels/train/train.json',
img_prefix='/content/data/images/train',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5),
dict(
type='OneOf',
transforms=[
dict(type='Identity'),
dict(type='AutoContrast'),
dict(type='RandEqualize'),
dict(type='RandSolarize'),
dict(type='RandColor'),
dict(type='RandContrast'),
dict(type='RandBrightness'),
dict(type='RandSharpness'),
dict(type='RandPosterize')
])
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='sup'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape',
'img_norm_cfg', 'pad_shape', 'scale_factor',
'tag'))
]),
unsup=dict(
type='CocoDataset',
ann_file='/content/data/labels/train/train.json',
img_prefix='/content/data/images/train',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='PseudoSamples', with_bbox=True),
dict(
type='MultiBranch',
unsup_student=[
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5),
dict(
type='ShuffledSequential',
transforms=[
dict(
type='OneOf',
transforms=[
dict(type='Identity'),
dict(type='AutoContrast'),
dict(type='RandEqualize'),
dict(type='RandSolarize'),
dict(type='RandColor'),
dict(type='RandContrast'),
dict(type='RandBrightness'),
dict(type='RandSharpness'),
dict(type='RandPosterize')
]),
dict(
type='OneOf',
transforms=[{
'type': 'RandTranslate',
'x': (-0.1, 0.1)
}, {
'type': 'RandTranslate',
'y': (-0.1, 0.1)
}, {
'type': 'RandRotate',
'angle': (-30, 30)
},
[{
'type':
'RandShear',
'x': (-30, 30)
}, {
'type':
'RandShear',
'y': (-30, 30)
}]])
]),
dict(
type='RandErase',
n_iterations=(1, 5),
size=[0, 0.2],
squared=True)
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='unsup_student'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape',
'img_norm_cfg', 'pad_shape',
'scale_factor', 'tag',
'transform_matrix'))
],
unsup_teacher=[
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5)
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='unsup_teacher'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape',
'img_norm_cfg', 'pad_shape',
'scale_factor', 'tag',
'transform_matrix'))
])
],
filter_empty_gt=False)),
val=dict(
type='CocoDataset',
ann_file='/content/data/labels/val/val.json',
img_prefix='/content/data/images/val',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]),
test=dict(
type='CocoDataset',
ann_file='/content/data/labels/test/test.json',
img_prefix='/content/data/images/test',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]),
sampler=dict(
train=dict(
type='SemiBalanceSampler',
sample_ratio=[1, 1],
by_prob=True,
epoch_length=200)))
evaluation = dict(interval=4000, metric='bbox', type='SubModulesDistEvalHook')
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=None)
lr_config = dict(
policy='step',
warmup='linear',
warmup_iters=500,
warmup_ratio=0.001,
step=[3000, 4000])
runner = dict(type='IterBasedRunner', max_iters=5000)
checkpoint_config = dict(interval=1000, by_epoch=False, max_keep_ckpts=2)
log_config = dict(
interval=50, hooks=[dict(type='TextLoggerHook', by_epoch=False)])
custom_hooks = [
dict(type='NumClassCheckHook'),
dict(type='WeightSummary'),
dict(type='MeanTeacher', momentum=0.999, interval=1, warm_up=0)
]
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
mmdet_base = '../../thirdparty/mmdetection/configs/_base_'
strong_pipeline = [
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5),
dict(
type='ShuffledSequential',
transforms=[
dict(
type='OneOf',
transforms=[
dict(type='Identity'),
dict(type='AutoContrast'),
dict(type='RandEqualize'),
dict(type='RandSolarize'),
dict(type='RandColor'),
dict(type='RandContrast'),
dict(type='RandBrightness'),
dict(type='RandSharpness'),
dict(type='RandPosterize')
]),
dict(
type='OneOf',
transforms=[{
'type': 'RandTranslate',
'x': (-0.1, 0.1)
}, {
'type': 'RandTranslate',
'y': (-0.1, 0.1)
}, {
'type': 'RandRotate',
'angle': (-30, 30)
},
[{
'type': 'RandShear',
'x': (-30, 30)
}, {
'type': 'RandShear',
'y': (-30, 30)
}]])
]),
dict(
type='RandErase',
n_iterations=(1, 5),
size=[0, 0.2],
squared=True)
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='unsup_student'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg',
'pad_shape', 'scale_factor', 'tag', 'transform_matrix'))
]
weak_pipeline = [
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5)
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='unsup_teacher'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg',
'pad_shape', 'scale_factor', 'tag', 'transform_matrix'))
]
unsup_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='PseudoSamples', with_bbox=True),
dict(
type='MultiBranch',
unsup_student=[
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5),
dict(
type='ShuffledSequential',
transforms=[
dict(
type='OneOf',
transforms=[
dict(type='Identity'),
dict(type='AutoContrast'),
dict(type='RandEqualize'),
dict(type='RandSolarize'),
dict(type='RandColor'),
dict(type='RandContrast'),
dict(type='RandBrightness'),
dict(type='RandSharpness'),
dict(type='RandPosterize')
]),
dict(
type='OneOf',
transforms=[{
'type': 'RandTranslate',
'x': (-0.1, 0.1)
}, {
'type': 'RandTranslate',
'y': (-0.1, 0.1)
}, {
'type': 'RandRotate',
'angle': (-30, 30)
},
[{
'type': 'RandShear',
'x': (-30, 30)
}, {
'type': 'RandShear',
'y': (-30, 30)
}]])
]),
dict(
type='RandErase',
n_iterations=(1, 5),
size=[0, 0.2],
squared=True)
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='unsup_student'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape',
'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag',
'transform_matrix'))
],
unsup_teacher=[
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5)
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='unsup_teacher'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape',
'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag',
'transform_matrix'))
])
]
fp16 = dict(loss_scale='dynamic')
percent = 10
fond = 1
work_dir = './work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k'
cfg_name = 'soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k'
gpu_ids = range(0, 1)
2021-11-02 09:31:10,382 - mmdet.ssod - INFO - initialize ResNet with init_cfg {'type': 'Pretrained', 'checkpoint': 'open-mmlab://detectron2/resnet50_caffe'}
2021-11-02 09:31:10,382 - mmcv - INFO - load model from: open-mmlab://detectron2/resnet50_caffe
2021-11-02 09:31:10,382 - mmcv - INFO - Use load_from_openmmlab loader
2021-11-02 09:31:10,459 - mmcv - WARNING - The model and loaded state dict do not match exactly
unexpected key in source state_dict: conv1.bias
2021-11-02 09:31:10,481 - mmdet.ssod - INFO - initialize FPN with init_cfg {'type': 'Xavier', 'layer': 'Conv2d', 'distribution': 'uniform'}
2021-11-02 09:31:10,504 - mmdet.ssod - INFO - initialize RPNHead with init_cfg {'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01}
2021-11-02 09:31:10,509 - mmdet.ssod - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}]
2021-11-02 09:31:10,632 - mmdet.ssod - INFO - initialize ResNet with init_cfg {'type': 'Pretrained', 'checkpoint': 'open-mmlab://detectron2/resnet50_caffe'}
2021-11-02 09:31:10,632 - mmcv - INFO - load model from: open-mmlab://detectron2/resnet50_caffe
2021-11-02 09:31:10,632 - mmcv - INFO - Use load_from_openmmlab loader
2021-11-02 09:31:10,688 - mmcv - WARNING - The model and loaded state dict do not match exactly
unexpected key in source state_dict: conv1.bias
2021-11-02 09:31:10,709 - mmdet.ssod - INFO - initialize FPN with init_cfg {'type': 'Xavier', 'layer': 'Conv2d', 'distribution': 'uniform'}
2021-11-02 09:31:10,731 - mmdet.ssod - INFO - initialize RPNHead with init_cfg {'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01}
2021-11-02 09:31:10,736 - mmdet.ssod - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}]
loading annotations into memory...
Done (t=0.03s)
creating index...
index created!
loading annotations into memory...
Done (t=0.03s)
creating index...
index created!
meta = {'env_info': 'sys.platform: linux\nPython: 3.7.12 (default, Sep 10 2021, 00:21:48) [GCC 7.5.0]\nCUDA available: True\nGPU 0: Tesla T4\nCUDA_HOME: /usr/local/cuda\nNVCC: Build cuda_11.1.TC455_06.29190527_0\nGCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0\nPyTorch: 1.9.0+cu111\nPyTorch compiling details: PyTorch built with:\n - GCC 7.3\n - C++ Version: 201402\n - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications\n - Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)\n - OpenMP 201511 (a.k.a. OpenMP 4.5)\n - NNPACK is enabled\n - CPU capability usage: AVX2\n - CUDA Runtime 11.1\n - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86\n - CuDNN 8.0.5\n - Magma 2.5.2\n - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, \n\nTorchVision: 0.10.0+cu111\nOpenCV: 4.1.2\nMMCV: 1.3.16\nMMCV Compiler: GCC 7.5\nMMCV CUDA Compiler: 11.1\nMMDetection: 2.18.0+2f5b0df', 'config': "model = dict(\n type='SoftTeacher',\n model=dict(\n type='FasterRCNN',\n backbone=dict(\n type='ResNet',\n depth=50,\n num_stages=4,\n out_indices=(0, 1, 2, 3),\n frozen_stages=1,\n norm_cfg=dict(type='BN', requires_grad=False),\n norm_eval=True,\n style='caffe',\n init_cfg=dict(\n type='Pretrained',\n checkpoint='open-mmlab://detectron2/resnet50_caffe')),\n neck=dict(\n type='FPN',\n in_channels=[256, 512, 1024, 2048],\n out_channels=256,\n num_outs=5),\n rpn_head=dict(\n type='RPNHead',\n in_channels=256,\n feat_channels=256,\n anchor_generator=dict(\n type='AnchorGenerator',\n scales=[8],\n ratios=[0.5, 1.0, 2.0],\n strides=[4, 8, 16, 32, 64]),\n bbox_coder=dict(\n type='DeltaXYWHBBoxCoder',\n target_means=[0.0, 0.0, 0.0, 0.0],\n target_stds=[1.0, 1.0, 1.0, 1.0]),\n loss_cls=dict(\n type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),\n loss_bbox=dict(type='L1Loss', loss_weight=1.0)),\n roi_head=dict(\n type='StandardRoIHead',\n bbox_roi_extractor=dict(\n type='SingleRoIExtractor',\n roi_layer=dict(\n type='RoIAlign', output_size=7, sampling_ratio=0),\n out_channels=256,\n featmap_strides=[4, 8, 16, 32]),\n bbox_head=dict(\n type='Shared2FCBBoxHead',\n in_channels=256,\n fc_out_channels=1024,\n roi_feat_size=7,\n num_classes=80,\n bbox_coder=dict(\n type='DeltaXYWHBBoxCoder',\n target_means=[0.0, 0.0, 0.0, 0.0],\n target_stds=[0.1, 0.1, 0.2, 0.2]),\n reg_class_agnostic=False,\n loss_cls=dict(\n type='CrossEntropyLoss',\n use_sigmoid=False,\n loss_weight=1.0),\n loss_bbox=dict(type='L1Loss', loss_weight=1.0))),\n train_cfg=dict(\n rpn=dict(\n assigner=dict(\n type='MaxIoUAssigner',\n pos_iou_thr=0.7,\n neg_iou_thr=0.3,\n min_pos_iou=0.3,\n match_low_quality=True,\n ignore_iof_thr=-1),\n sampler=dict(\n type='RandomSampler',\n num=256,\n pos_fraction=0.5,\n neg_pos_ub=-1,\n add_gt_as_proposals=False),\n allowed_border=-1,\n pos_weight=-1,\n debug=False),\n rpn_proposal=dict(\n nms_pre=2000,\n max_per_img=1000,\n nms=dict(type='nms', iou_threshold=0.7),\n min_bbox_size=0),\n rcnn=dict(\n assigner=dict(\n type='MaxIoUAssigner',\n pos_iou_thr=0.5,\n neg_iou_thr=0.5,\n min_pos_iou=0.5,\n match_low_quality=False,\n ignore_iof_thr=-1),\n sampler=dict(\n type='RandomSampler',\n num=512,\n pos_fraction=0.25,\n neg_pos_ub=-1,\n add_gt_as_proposals=True),\n pos_weight=-1,\n debug=False)),\n test_cfg=dict(\n rpn=dict(\n nms_pre=1000,\n max_per_img=1000,\n nms=dict(type='nms', iou_threshold=0.7),\n min_bbox_size=0),\n rcnn=dict(\n score_thr=0.05,\n nms=dict(type='nms', iou_threshold=0.5),\n max_per_img=100))),\n train_cfg=dict(\n use_teacher_proposal=False,\n pseudo_label_initial_score_thr=0.5,\n rpn_pseudo_threshold=0.9,\n cls_pseudo_threshold=0.9,\n reg_pseudo_threshold=0.02,\n jitter_times=10,\n jitter_scale=0.06,\n min_pseduo_box_size=0,\n unsup_weight=2.0),\n test_cfg=dict(inference_on='student'))\ndataset_type = 'CocoDataset'\ndata_root = '/content/data/'\nimg_norm_cfg = dict(\n mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False)\ntrain_pipeline = [\n dict(type='LoadImageFromFile'),\n dict(type='LoadAnnotations', with_bbox=True),\n dict(\n type='Sequential',\n transforms=[\n dict(\n type='RandResize',\n img_scale=[(1333, 400), (1333, 1200)],\n multiscale_mode='range',\n keep_ratio=True),\n dict(type='RandFlip', flip_ratio=0.5),\n dict(\n type='OneOf',\n transforms=[\n dict(type='Identity'),\n dict(type='AutoContrast'),\n dict(type='RandEqualize'),\n dict(type='RandSolarize'),\n dict(type='RandColor'),\n dict(type='RandContrast'),\n dict(type='RandBrightness'),\n dict(type='RandSharpness'),\n dict(type='RandPosterize')\n ])\n ],\n record=True),\n dict(type='Pad', size_divisor=32),\n dict(\n type='Normalize',\n mean=[103.53, 116.28, 123.675],\n std=[1.0, 1.0, 1.0],\n to_rgb=False),\n dict(type='ExtraAttrs', tag='sup'),\n dict(type='DefaultFormatBundle'),\n dict(\n type='Collect',\n keys=['img', 'gt_bboxes', 'gt_labels'],\n meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg',\n 'pad_shape', 'scale_factor', 'tag'))\n]\ntest_pipeline = [\n dict(type='LoadImageFromFile'),\n dict(\n type='MultiScaleFlipAug',\n img_scale=(1333, 800),\n flip=False,\n transforms=[\n dict(type='Resize', keep_ratio=True),\n dict(type='RandomFlip'),\n dict(\n type='Normalize',\n mean=[103.53, 116.28, 123.675],\n std=[1.0, 1.0, 1.0],\n to_rgb=False),\n dict(type='Pad', size_divisor=32),\n dict(type='ImageToTensor', keys=['img']),\n dict(type='Collect', keys=['img'])\n ])\n]\ndata = dict(\n samples_per_gpu=1,\n workers_per_gpu=2,\n train=dict(\n type='SemiDataset',\n sup=dict(\n type='CocoDataset',\n ann_file='/content/data/labels/train/train.json',\n img_prefix='/content/data/images/train',\n pipeline=[\n dict(type='LoadImageFromFile'),\n dict(type='LoadAnnotations', with_bbox=True),\n dict(\n type='Sequential',\n transforms=[\n dict(\n type='RandResize',\n img_scale=[(1333, 400), (1333, 1200)],\n multiscale_mode='range',\n keep_ratio=True),\n dict(type='RandFlip', flip_ratio=0.5),\n dict(\n type='OneOf',\n transforms=[\n dict(type='Identity'),\n dict(type='AutoContrast'),\n dict(type='RandEqualize'),\n dict(type='RandSolarize'),\n dict(type='RandColor'),\n dict(type='RandContrast'),\n dict(type='RandBrightness'),\n dict(type='RandSharpness'),\n dict(type='RandPosterize')\n ])\n ],\n record=True),\n dict(type='Pad', size_divisor=32),\n dict(\n type='Normalize',\n mean=[103.53, 116.28, 123.675],\n std=[1.0, 1.0, 1.0],\n to_rgb=False),\n dict(type='ExtraAttrs', tag='sup'),\n dict(type='DefaultFormatBundle'),\n dict(\n type='Collect',\n keys=['img', 'gt_bboxes', 'gt_labels'],\n meta_keys=('filename', 'ori_shape', 'img_shape',\n 'img_norm_cfg', 'pad_shape', 'scale_factor',\n 'tag'))\n ]),\n unsup=dict(\n type='CocoDataset',\n ann_file='/content/data/labels/train/train.json',\n img_prefix='/content/data/images/train',\n pipeline=[\n dict(type='LoadImageFromFile'),\n dict(type='PseudoSamples', with_bbox=True),\n dict(\n type='MultiBranch',\n unsup_student=[\n dict(\n type='Sequential',\n transforms=[\n dict(\n type='RandResize',\n img_scale=[(1333, 400), (1333, 1200)],\n multiscale_mode='range',\n keep_ratio=True),\n dict(type='RandFlip', flip_ratio=0.5),\n dict(\n type='ShuffledSequential',\n transforms=[\n dict(\n type='OneOf',\n transforms=[\n dict(type='Identity'),\n dict(type='AutoContrast'),\n dict(type='RandEqualize'),\n dict(type='RandSolarize'),\n dict(type='RandColor'),\n dict(type='RandContrast'),\n dict(type='RandBrightness'),\n dict(type='RandSharpness'),\n dict(type='RandPosterize')\n ]),\n dict(\n type='OneOf',\n transforms=[{\n 'type': 'RandTranslate',\n 'x': (-0.1, 0.1)\n }, {\n 'type': 'RandTranslate',\n 'y': (-0.1, 0.1)\n }, {\n 'type': 'RandRotate',\n 'angle': (-30, 30)\n },\n [{\n 'type':\n 'RandShear',\n 'x': (-30, 30)\n }, {\n 'type':\n 'RandShear',\n 'y': (-30, 30)\n }]])\n ]),\n dict(\n type='RandErase',\n n_iterations=(1, 5),\n size=[0, 0.2],\n squared=True)\n ],\n record=True),\n dict(type='Pad', size_divisor=32),\n dict(\n type='Normalize',\n mean=[103.53, 116.28, 123.675],\n std=[1.0, 1.0, 1.0],\n to_rgb=False),\n dict(type='ExtraAttrs', tag='unsup_student'),\n dict(type='DefaultFormatBundle'),\n dict(\n type='Collect',\n keys=['img', 'gt_bboxes', 'gt_labels'],\n meta_keys=('filename', 'ori_shape', 'img_shape',\n 'img_norm_cfg', 'pad_shape',\n 'scale_factor', 'tag',\n 'transform_matrix'))\n ],\n unsup_teacher=[\n dict(\n type='Sequential',\n transforms=[\n dict(\n type='RandResize',\n img_scale=[(1333, 400), (1333, 1200)],\n multiscale_mode='range',\n keep_ratio=True),\n dict(type='RandFlip', flip_ratio=0.5)\n ],\n record=True),\n dict(type='Pad', size_divisor=32),\n dict(\n type='Normalize',\n mean=[103.53, 116.28, 123.675],\n std=[1.0, 1.0, 1.0],\n to_rgb=False),\n dict(type='ExtraAttrs', tag='unsup_teacher'),\n dict(type='DefaultFormatBundle'),\n dict(\n type='Collect',\n keys=['img', 'gt_bboxes', 'gt_labels'],\n meta_keys=('filename', 'ori_shape', 'img_shape',\n 'img_norm_cfg', 'pad_shape',\n 'scale_factor', 'tag',\n 'transform_matrix'))\n ])\n ],\n filter_empty_gt=False)),\n val=dict(\n type='CocoDataset',\n ann_file='/content/data/labels/val/val.json',\n img_prefix='/content/data/images/val',\n pipeline=[\n dict(type='LoadImageFromFile'),\n dict(\n type='MultiScaleFlipAug',\n img_scale=(1333, 800),\n flip=False,\n transforms=[\n dict(type='Resize', keep_ratio=True),\n dict(type='RandomFlip'),\n dict(\n type='Normalize',\n mean=[103.53, 116.28, 123.675],\n std=[1.0, 1.0, 1.0],\n to_rgb=False),\n dict(type='Pad', size_divisor=32),\n dict(type='ImageToTensor', keys=['img']),\n dict(type='Collect', keys=['img'])\n ])\n ]),\n test=dict(\n type='CocoDataset',\n ann_file='/content/data/labels/test/test.json',\n img_prefix='/content/data/images/test',\n pipeline=[\n dict(type='LoadImageFromFile'),\n dict(\n type='MultiScaleFlipAug',\n img_scale=(1333, 800),\n flip=False,\n transforms=[\n dict(type='Resize', keep_ratio=True),\n dict(type='RandomFlip'),\n dict(\n type='Normalize',\n mean=[103.53, 116.28, 123.675],\n std=[1.0, 1.0, 1.0],\n to_rgb=False),\n dict(type='Pad', size_divisor=32),\n dict(type='ImageToTensor', keys=['img']),\n dict(type='Collect', keys=['img'])\n ])\n ]),\n sampler=dict(\n train=dict(\n type='SemiBalanceSampler',\n sample_ratio=[1, 1],\n by_prob=True,\n epoch_length=200)))\nevaluation = dict(interval=4000, metric='bbox', type='SubModulesDistEvalHook')\noptimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)\noptimizer_config = dict(grad_clip=None)\nlr_config = dict(\n policy='step',\n warmup='linear',\n warmup_iters=500,\n warmup_ratio=0.001,\n step=[3000, 4000])\nrunner = dict(type='IterBasedRunner', max_iters=5000)\ncheckpoint_config = dict(interval=1000, by_epoch=False, max_keep_ckpts=2)\nlog_config = dict(\n interval=50, hooks=[dict(type='TextLoggerHook', by_epoch=False)])\ncustom_hooks = [\n dict(type='NumClassCheckHook'),\n dict(type='WeightSummary'),\n dict(type='MeanTeacher', momentum=0.999, interval=1, warm_up=0)\n]\ndist_params = dict(backend='nccl')\nlog_level = 'INFO'\nload_from = None\nresume_from = None\nworkflow = [('train', 1)]\nmmdet_base = '../../thirdparty/mmdetection/configs/_base_'\nstrong_pipeline = [\n dict(\n type='Sequential',\n transforms=[\n dict(\n type='RandResize',\n img_scale=[(1333, 400), (1333, 1200)],\n multiscale_mode='range',\n keep_ratio=True),\n dict(type='RandFlip', flip_ratio=0.5),\n dict(\n type='ShuffledSequential',\n transforms=[\n dict(\n type='OneOf',\n transforms=[\n dict(type='Identity'),\n dict(type='AutoContrast'),\n dict(type='RandEqualize'),\n dict(type='RandSolarize'),\n dict(type='RandColor'),\n dict(type='RandContrast'),\n dict(type='RandBrightness'),\n dict(type='RandSharpness'),\n dict(type='RandPosterize')\n ]),\n dict(\n type='OneOf',\n transforms=[{\n 'type': 'RandTranslate',\n 'x': (-0.1, 0.1)\n }, {\n 'type': 'RandTranslate',\n 'y': (-0.1, 0.1)\n }, {\n 'type': 'RandRotate',\n 'angle': (-30, 30)\n },\n [{\n 'type': 'RandShear',\n 'x': (-30, 30)\n }, {\n 'type': 'RandShear',\n 'y': (-30, 30)\n }]])\n ]),\n dict(\n type='RandErase',\n n_iterations=(1, 5),\n size=[0, 0.2],\n squared=True)\n ],\n record=True),\n dict(type='Pad', size_divisor=32),\n dict(\n type='Normalize',\n mean=[103.53, 116.28, 123.675],\n std=[1.0, 1.0, 1.0],\n to_rgb=False),\n dict(type='ExtraAttrs', tag='unsup_student'),\n dict(type='DefaultFormatBundle'),\n dict(\n type='Collect',\n keys=['img', 'gt_bboxes', 'gt_labels'],\n meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg',\n 'pad_shape', 'scale_factor', 'tag', 'transform_matrix'))\n]\nweak_pipeline = [\n dict(\n type='Sequential',\n transforms=[\n dict(\n type='RandResize',\n img_scale=[(1333, 400), (1333, 1200)],\n multiscale_mode='range',\n keep_ratio=True),\n dict(type='RandFlip', flip_ratio=0.5)\n ],\n record=True),\n dict(type='Pad', size_divisor=32),\n dict(\n type='Normalize',\n mean=[103.53, 116.28, 123.675],\n std=[1.0, 1.0, 1.0],\n to_rgb=False),\n dict(type='ExtraAttrs', tag='unsup_teacher'),\n dict(type='DefaultFormatBundle'),\n dict(\n type='Collect',\n keys=['img', 'gt_bboxes', 'gt_labels'],\n meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg',\n 'pad_shape', 'scale_factor', 'tag', 'transform_matrix'))\n]\nunsup_pipeline = [\n dict(type='LoadImageFromFile'),\n dict(type='PseudoSamples', with_bbox=True),\n dict(\n type='MultiBranch',\n unsup_student=[\n dict(\n type='Sequential',\n transforms=[\n dict(\n type='RandResize',\n img_scale=[(1333, 400), (1333, 1200)],\n multiscale_mode='range',\n keep_ratio=True),\n dict(type='RandFlip', flip_ratio=0.5),\n dict(\n type='ShuffledSequential',\n transforms=[\n dict(\n type='OneOf',\n transforms=[\n dict(type='Identity'),\n dict(type='AutoContrast'),\n dict(type='RandEqualize'),\n dict(type='RandSolarize'),\n dict(type='RandColor'),\n dict(type='RandContrast'),\n dict(type='RandBrightness'),\n dict(type='RandSharpness'),\n dict(type='RandPosterize')\n ]),\n dict(\n type='OneOf',\n transforms=[{\n 'type': 'RandTranslate',\n 'x': (-0.1, 0.1)\n }, {\n 'type': 'RandTranslate',\n 'y': (-0.1, 0.1)\n }, {\n 'type': 'RandRotate',\n 'angle': (-30, 30)\n },\n [{\n 'type': 'RandShear',\n 'x': (-30, 30)\n }, {\n 'type': 'RandShear',\n 'y': (-30, 30)\n }]])\n ]),\n dict(\n type='RandErase',\n n_iterations=(1, 5),\n size=[0, 0.2],\n squared=True)\n ],\n record=True),\n dict(type='Pad', size_divisor=32),\n dict(\n type='Normalize',\n mean=[103.53, 116.28, 123.675],\n std=[1.0, 1.0, 1.0],\n to_rgb=False),\n dict(type='ExtraAttrs', tag='unsup_student'),\n dict(type='DefaultFormatBundle'),\n dict(\n type='Collect',\n keys=['img', 'gt_bboxes', 'gt_labels'],\n meta_keys=('filename', 'ori_shape', 'img_shape',\n 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag',\n 'transform_matrix'))\n ],\n unsup_teacher=[\n dict(\n type='Sequential',\n transforms=[\n dict(\n type='RandResize',\n img_scale=[(1333, 400), (1333, 1200)],\n multiscale_mode='range',\n keep_ratio=True),\n dict(type='RandFlip', flip_ratio=0.5)\n ],\n record=True),\n dict(type='Pad', size_divisor=32),\n dict(\n type='Normalize',\n mean=[103.53, 116.28, 123.675],\n std=[1.0, 1.0, 1.0],\n to_rgb=False),\n dict(type='ExtraAttrs', tag='unsup_teacher'),\n dict(type='DefaultFormatBundle'),\n dict(\n type='Collect',\n keys=['img', 'gt_bboxes', 'gt_labels'],\n meta_keys=('filename', 'ori_shape', 'img_shape',\n 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag',\n 'transform_matrix'))\n ])\n]\nfp16 = dict(loss_scale='dynamic')\npercent = 10\nfond = 1\nwork_dir = './work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k'\ncfg_name = 'soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k'\ngpu_ids = range(0, 1)\n", 'seed': None, 'exp_name': 'soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py'}
build sampler = SemiBalanceSampler
group = False
dist = False
end sampler_type = SemiBalanceSampler
Traceback (most recent call last):
File "train.py", line 199, in <module>
main()
File "train.py", line 194, in main
meta=meta,
File "/content/SoftTeacher/ssod/apis/train.py", line 81, in train_detector
for ds in dataset
File "/content/SoftTeacher/ssod/apis/train.py", line 81, in <listcomp>
for ds in dataset
File "/content/SoftTeacher/ssod/datasets/builder.py", line 74, in build_dataloader
if shuffle
File "/content/SoftTeacher/ssod/datasets/builder.py", line 45, in build_sampler
return build_from_cfg(cfg, SAMPLERS, default_args)
File "/usr/local/lib/python3.7/dist-packages/mmcv/utils/registry.py", line 45, in build_from_cfg
f'{obj_type} is not in the {registry.name} registry')
KeyError: 'SemiBalanceSampler is not in the sampler registry'
When i run this:
CFG_PATH=$'/content/SoftTeacher/configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py'
PORT=${PORT:-29500}
!python -m torch.distributed.launch --nproc_per_node=1 --master_port=$PORT \
$(dirname "$0")/train.py $CFG_PATH --launcher pytorch ${@:3}
I get error:
Traceback (most recent call last):
File "./train.py", line 199, in <module>
main()
File "./train.py", line 194, in main
meta=meta,
File "/content/SoftTeacher/ssod/apis/train.py", line 206, in train_detector
runner.run(data_loaders, cfg.workflow)
File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/iter_based_runner.py", line 117, in run
iter_loaders = [IterLoader(x) for x in data_loaders]
File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/iter_based_runner.py", line 117, in <listcomp>
iter_loaders = [IterLoader(x) for x in data_loaders]
File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/iter_based_runner.py", line 23, in __init__
self.iter_loader = iter(self._dataloader)
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 359, in __iter__
return self._get_iterator()
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 305, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 944, in __init__
self._reset(loader, first_iter=True)
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 975, in _reset
self._try_put_index()
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1209, in _try_put_index
index = self._next_index()
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 512, in _next_index
return next(self._sampler_iter) # may raise StopIteration
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/sampler.py", line 226, in __iter__
for idx in self.sampler:
File "/content/SoftTeacher/ssod/datasets/samplers/semi_sampler.py", line 190, in __iter__
assert len(indices) == len(self)
AssertionError
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 5481) of binary: /usr/bin/python3
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
Okey, i dont understand. I use the script tools/dist_train.sh. I want to train single GPU
My config file:
_base_="base.py"
data = dict(
samples_per_gpu=1,
workers_per_gpu=1,
train=dict(
sup=dict(
ann_file="/content/semi_data/sup/annotations/annotation.json",
img_prefix="/content/semi_data/sup/image",
),
unsup=dict(
ann_file="/content/semi_data/unsup/annotations/annotation.json",
img_prefix="/content/semi_data/unsup/image",
),
),
sampler=dict(
train=dict(
sample_ratio=[1, 1],
)
),
)
semi_wrapper = dict(
train_cfg=dict(
unsup_weight=2.0,
)
)
evaluation = dict(interval=1000, metric='bbox', type='SubModulesDistEvalHook')
optimizer = dict(type='SGD', lr=0.001, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=None)
evaluation = dict(type="SubModulesDistEvalHook", interval=4000)
optimizer = dict(type="SGD", lr=0.01, momentum=0.9, weight_decay=0.0001)
lr_config = dict(step=[3000, 4000])
runner = dict(_delete_=True, type="IterBasedRunner", max_iters=5000)
checkpoint_config = dict(by_epoch=False, interval=1000, max_keep_ckpts=2)
And i change epoch_length in base.py file, i set 1000 But i get error:
Traceback (most recent call last):
File "./train.py", line 201, in <module>
main()
File "./train.py", line 196, in main
meta=meta,
File "/content/SoftTeacher/ssod/apis/train.py", line 206, in train_detector
runner.run(data_loaders, cfg.workflow)
File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/iter_based_runner.py", line 117, in run
iter_loaders = [IterLoader(x) for x in data_loaders]
File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/iter_based_runner.py", line 117, in <listcomp>
iter_loaders = [IterLoader(x) for x in data_loaders]
File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/iter_based_runner.py", line 23, in __init__
self.iter_loader = iter(self._dataloader)
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 359, in __iter__
return self._get_iterator()
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 305, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 944, in __init__
self._reset(loader, first_iter=True)
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 975, in _reset
self._try_put_index()
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1209, in _try_put_index
index = self._next_index()
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 512, in _next_index
return next(self._sampler_iter) # may raise StopIteration
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/sampler.py", line 226, in __iter__
for idx in self.sampler:
File "/content/SoftTeacher/ssod/datasets/samplers/semi_sampler.py", line 195, in __iter__
assert len(indices) == len(self)
I think the problem is your samples_per_gpu. The sampler is designed to sample at least one sample per group (sup and unsup) per gpu.
@MendelXu I just set by_prob=False, and it work. But i get new error: assert len(cls_scores) == len(score_factors) AssertionError
I got the similar result. I run the code in local linux server:
python tools/train.py configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_180k.py --gpus 2 --cfg-options fold=1 percent=1
However
Traceback (most recent call last): File "tools/train.py", line 198, in
main() File "tools/train.py", line 193, in main meta=meta, File "/GPFS/liying_lab_permanent/XXXX/SoftTeacher/ssod/apis/train.py", line 81, in train_detector for ds in dataset File "/GPFS/liying_lab_permanent/XXXX/SoftTeacher/ssod/apis/train.py", line 81, in for ds in dataset File "/GPFS/liying_lab_permanent/XXXX/SoftTeacher/ssod/datasets/builder.py", line 69, in build_dataloader if shuffle File "/GPFS/liying_lab_permanent/XXXX/SoftTeacher/ssod/datasets/builder.py", line 40, in build_sampler return build_from_cfg(cfg, SAMPLERS, default_args) File "/home/liying_lab/XXXX/.conda/envs/open-mmlab/lib/python3.7/site-packages/mmcv/utils/registry.py", line 44, in build_from_cfg f'{obj_type} is not in the {registry.name} registry') KeyError: 'SemiBalanceSampler is not in the sampler registry'
KeyError: 'SemiBalanceSampler is not in the sampler registry'.
Indeed, there is no SemiBalanceSampler in the code at all. find . -name \*.py | xargs grep SemiBalanceSampler
./work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_180k/1/1/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_180k.py:397: type='SemiBalanceSampler', ./ssod/datasets/init.py:7:from .samplers import DistributedGroupSemiBalanceSampler ./ssod/datasets/init.py:14: "DistributedGroupSemiBalanceSampler", ./ssod/datasets/samplers/semi_sampler.py:12:class DistributedGroupSemiBalanceSampler(Sampler): ./ssod/datasets/samplers/init.py:1:from .semi_sampler import DistributedGroupSemiBalanceSampler ./ssod/datasets/samplers/init.py:3: "DistributedGroupSemiBalanceSampler", ./configs/soft_teacher/base.py:230: type="SemiBalanceSampler",
@MendelXu I just set by_prob=False, and it work. But i get new error: assert len(cls_scores) == len(score_factors) AssertionError
I have no idea currently. I will try to debug it tomarrow.
@MendelXu I just set by_prob=False, and it work. But i get new error: assert len(cls_scores) == len(score_factors) AssertionError You can try this: https://github.com/microsoft/SoftTeacher/issues/83#issuecomment-955626134
I got the same error, how to fix it? thanks!
@MendelXu I just set by_prob=False, and it work. But i get new error: assert len(cls_scores) == len(score_factors) AssertionError
by_prob = False also solved the assert len(indices) = len(self), however I've got a new problem now.
I have been here before and I was advised to update nccl, I did with pytorch 1.10.0 and got nccl v2.10.3. Still getting error similar to the initial one:
/home/lfgp/miniconda3/lib/python3.9/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See
https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions
warnings.warn(
2021-11-10 14:18:12,285 - mmdet.ssod - INFO - [<StreamHandler <stderr> (INFO)>, <FileHandler /mnt/c/Users/Francisco Pereira/Desktop/IST/10º semestre/algoritmos/SoftTeacher/work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k/20211110_141812.log (INFO)>]
2021-11-10 14:18:12,286 - mmdet.ssod - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.9.5 (default, Jun 4 2021, 12:28:51) [GCC 7.5.0]
CUDA available: True
GPU 0: NVIDIA GeForce GTX 1650 with Max-Q Design
CUDA_HOME: None
GCC: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
PyTorch: 1.10.0
PyTorch compiling details: PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 10.2
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
- CuDNN 7.6.5
- Magma 2.5.2
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
TorchVision: 0.11.1
OpenCV: 4.5.4-dev
MMCV: 1.3.16
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 10.2
MMDetection: 2.17.0+944b131
------------------------------------------------------------
2021-11-10 14:18:15,033 - mmdet.ssod - INFO - Distributed training: True
2021-11-10 14:18:17,923 - mmdet.ssod - INFO - Config:
model = dict(
type='SoftTeacher',
model=dict(
type='FasterRCNN',
backbone=dict(
type='ResNet',
depth=50,
num_stages=4,
out_indices=(0, 1, 2, 3),
frozen_stages=1,
norm_cfg=dict(type='BN', requires_grad=False),
norm_eval=True,
style='caffe',
init_cfg=dict(
type='Pretrained',
checkpoint='open-mmlab://detectron2/resnet50_caffe')),
neck=dict(
type='FPN',
in_channels=[256, 512, 1024, 2048],
out_channels=256,
num_outs=5),
rpn_head=dict(
type='RPNHead',
in_channels=256,
feat_channels=256,
anchor_generator=dict(
type='AnchorGenerator',
scales=[8],
ratios=[0.5, 1.0, 2.0],
strides=[4, 8, 16, 32, 64]),
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[1.0, 1.0, 1.0, 1.0]),
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0)),
roi_head=dict(
type='StandardRoIHead',
bbox_roi_extractor=dict(
type='SingleRoIExtractor',
roi_layer=dict(
type='RoIAlign', output_size=7, sampling_ratio=0),
out_channels=256,
featmap_strides=[4, 8, 16, 32]),
bbox_head=dict(
type='Shared2FCBBoxHead',
in_channels=256,
fc_out_channels=1024,
roi_feat_size=7,
num_classes=80,
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[0.1, 0.1, 0.2, 0.2]),
reg_class_agnostic=False,
loss_cls=dict(
type='CrossEntropyLoss',
use_sigmoid=False,
loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0))),
train_cfg=dict(
rpn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.7,
neg_iou_thr=0.3,
min_pos_iou=0.3,
match_low_quality=True,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=256,
pos_fraction=0.5,
neg_pos_ub=-1,
add_gt_as_proposals=False),
allowed_border=-1,
pos_weight=-1,
debug=False),
rpn_proposal=dict(
nms_pre=2000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.5,
neg_iou_thr=0.5,
min_pos_iou=0.5,
match_low_quality=False,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=512,
pos_fraction=0.25,
neg_pos_ub=-1,
add_gt_as_proposals=True),
pos_weight=-1,
debug=False)),
test_cfg=dict(
rpn=dict(
nms_pre=1000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
score_thr=0.05,
nms=dict(type='nms', iou_threshold=0.5),
max_per_img=100))),
train_cfg=dict(
use_teacher_proposal=False,
pseudo_label_initial_score_thr=0.5,
rpn_pseudo_threshold=0.9,
cls_pseudo_threshold=0.9,
reg_pseudo_threshold=0.02,
jitter_times=10,
jitter_scale=0.06,
min_pseduo_box_size=0,
unsup_weight=2.0),
test_cfg=dict(inference_on='student'))
dataset_type = 'CocoDataset'
data_root = 'data/coco/'
img_norm_cfg = dict(
mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False)
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5),
dict(
type='OneOf',
transforms=[
dict(type='Identity'),
dict(type='AutoContrast'),
dict(type='RandEqualize'),
dict(type='RandSolarize'),
dict(type='RandColor'),
dict(type='RandContrast'),
dict(type='RandBrightness'),
dict(type='RandSharpness'),
dict(type='RandPosterize')
])
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='sup'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg',
'pad_shape', 'scale_factor', 'tag'))
]
test_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]
data = dict(
samples_per_gpu=8,
workers_per_gpu=8,
train=dict(
type='SemiDataset',
sup=dict(
type='MyDataset',
ann_file='data/coco/annotations/instances_train2017.json',
img_prefix='data/coco/train2017/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5),
dict(
type='OneOf',
transforms=[
dict(type='Identity'),
dict(type='AutoContrast'),
dict(type='RandEqualize'),
dict(type='RandSolarize'),
dict(type='RandColor'),
dict(type='RandContrast'),
dict(type='RandBrightness'),
dict(type='RandSharpness'),
dict(type='RandPosterize')
])
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='sup'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape',
'img_norm_cfg', 'pad_shape', 'scale_factor',
'tag'))
]),
unsup=dict(
type='MyDataset',
ann_file='data/coco/annotations/instances_unlabeled2017.json',
img_prefix='data/coco/unlabeled2017/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='PseudoSamples', with_bbox=True),
dict(
type='MultiBranch',
unsup_student=[
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5),
dict(
type='ShuffledSequential',
transforms=[
dict(
type='OneOf',
transforms=[
dict(type='Identity'),
dict(type='AutoContrast'),
dict(type='RandEqualize'),
dict(type='RandSolarize'),
dict(type='RandColor'),
dict(type='RandContrast'),
dict(type='RandBrightness'),
dict(type='RandSharpness'),
dict(type='RandPosterize')
]),
dict(
type='OneOf',
transforms=[{
'type': 'RandTranslate',
'x': (-0.1, 0.1)
}, {
'type': 'RandTranslate',
'y': (-0.1, 0.1)
}, {
'type': 'RandRotate',
'angle': (-30, 30)
},
[{
'type':
'RandShear',
'x': (-30, 30)
}, {
'type':
'RandShear',
'y': (-30, 30)
}]])
]),
dict(
type='RandErase',
n_iterations=(1, 5),
size=[0, 0.2],
squared=True)
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='unsup_student'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape',
'img_norm_cfg', 'pad_shape',
'scale_factor', 'tag',
'transform_matrix'))
],
unsup_teacher=[
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5)
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='unsup_teacher'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape',
'img_norm_cfg', 'pad_shape',
'scale_factor', 'tag',
'transform_matrix'))
])
],
filter_empty_gt=False)),
val=dict(
type='CocoDataset',
ann_file='data/coco/annotations/instances_val2017.json',
img_prefix='data/coco/val2017/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]),
test=dict(
type='CocoDataset',
ann_file='data/coco/annotations/instances_val2017.json',
img_prefix='data/coco/val2017/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]),
sampler=dict(
train=dict(
type='SemiBalanceSampler',
sample_ratio=[1, 1],
by_prob=False,
epoch_length=7330)))
evaluation = dict(interval=4000, metric='bbox', type='SubModulesDistEvalHook')
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=None)
lr_config = dict(
policy='step',
warmup='linear',
warmup_iters=500,
warmup_ratio=0.001,
step=[480000, 640000])
runner = dict(type='IterBasedRunner', max_iters=720000)
checkpoint_config = dict(interval=4000, by_epoch=False, max_keep_ckpts=20)
log_config = dict(
interval=50,
hooks=[
dict(type='TextLoggerHook', by_epoch=False),
dict(
type='WandbLoggerHook',
init_kwargs=dict(
project='pre_release',
name='soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k',
config=dict(
work_dirs=
'./work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k',
total_step=720000)),
by_epoch=False)
])
custom_hooks = [
dict(type='NumClassCheckHook'),
dict(type='WeightSummary'),
dict(type='MeanTeacher', momentum=0.999, interval=1, warm_up=0)
]
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
mmdet_base = '../../thirdparty/mmdetection/configs/_base_'
strong_pipeline = [
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5),
dict(
type='ShuffledSequential',
transforms=[
dict(
type='OneOf',
transforms=[
dict(type='Identity'),
dict(type='AutoContrast'),
dict(type='RandEqualize'),
dict(type='RandSolarize'),
dict(type='RandColor'),
dict(type='RandContrast'),
dict(type='RandBrightness'),
dict(type='RandSharpness'),
dict(type='RandPosterize')
]),
dict(
type='OneOf',
transforms=[{
'type': 'RandTranslate',
'x': (-0.1, 0.1)
}, {
'type': 'RandTranslate',
'y': (-0.1, 0.1)
}, {
'type': 'RandRotate',
'angle': (-30, 30)
},
[{
'type': 'RandShear',
'x': (-30, 30)
}, {
'type': 'RandShear',
'y': (-30, 30)
}]])
]),
dict(
type='RandErase',
n_iterations=(1, 5),
size=[0, 0.2],
squared=True)
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='unsup_student'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg',
'pad_shape', 'scale_factor', 'tag', 'transform_matrix'))
]
weak_pipeline = [
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5)
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='unsup_teacher'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg',
'pad_shape', 'scale_factor', 'tag', 'transform_matrix'))
]
unsup_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='PseudoSamples', with_bbox=True),
dict(
type='MultiBranch',
unsup_student=[
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5),
dict(
type='ShuffledSequential',
transforms=[
dict(
type='OneOf',
transforms=[
dict(type='Identity'),
dict(type='AutoContrast'),
dict(type='RandEqualize'),
dict(type='RandSolarize'),
dict(type='RandColor'),
dict(type='RandContrast'),
dict(type='RandBrightness'),
dict(type='RandSharpness'),
dict(type='RandPosterize')
]),
dict(
type='OneOf',
transforms=[{
'type': 'RandTranslate',
'x': (-0.1, 0.1)
}, {
'type': 'RandTranslate',
'y': (-0.1, 0.1)
}, {
'type': 'RandRotate',
'angle': (-30, 30)
},
[{
'type': 'RandShear',
'x': (-30, 30)
}, {
'type': 'RandShear',
'y': (-30, 30)
}]])
]),
dict(
type='RandErase',
n_iterations=(1, 5),
size=[0, 0.2],
squared=True)
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='unsup_student'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape',
'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag',
'transform_matrix'))
],
unsup_teacher=[
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5)
],
record=True),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='unsup_teacher'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape',
'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag',
'transform_matrix'))
])
]
fp16 = dict(loss_scale='dynamic')
work_dir = './work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k'
cfg_name = 'soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k'
gpu_ids = range(0, 1)
/mnt/c/Users/Francisco Pereira/Desktop/IST/10º semestre/algoritmos/SoftTeacher/thirdparty/mmdetection/mmdet/core/anchor/builder.py:16: UserWarning: ``build_anchor_generator`` would be deprecated soon, please use ``build_prior_generator``
warnings.warn(
2021-11-10 14:11:18,402 - mmdet.ssod - INFO - initialize ResNet with init_cfg {'type': 'Pretrained', 'checkpoint': 'open-mmlab://detectron2/resnet50_caffe'}
2021-11-10 14:11:18,403 - mmcv - INFO - load model from: open-mmlab://detectron2/resnet50_caffe
2021-11-10 14:11:18,403 - mmcv - INFO - Use load_from_openmmlab loader
2021-11-10 14:11:18,535 - mmcv - WARNING - The model and loaded state dict do not match exactly
unexpected key in source state_dict: conv1.bias
2021-11-10 14:11:18,560 - mmdet.ssod - INFO - initialize FPN with init_cfg {'type': 'Xavier', 'layer': 'Conv2d', 'distribution': 'uniform'}
2021-11-10 14:11:18,581 - mmdet.ssod - INFO - initialize RPNHead with init_cfg {'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01}
2021-11-10 14:11:18,587 - mmdet.ssod - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'layer': 'Linear', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}]
2021-11-10 14:11:18,791 - mmdet.ssod - INFO - initialize ResNet with init_cfg {'type': 'Pretrained', 'checkpoint': 'open-mmlab://detectron2/resnet50_caffe'}
2021-11-10 14:11:18,792 - mmcv - INFO - load model from: open-mmlab://detectron2/resnet50_caffe
2021-11-10 14:11:18,792 - mmcv - INFO - Use load_from_openmmlab loader
2021-11-10 14:11:18,859 - mmcv - WARNING - The model and loaded state dict do not match exactly
unexpected key in source state_dict: conv1.bias
2021-11-10 14:11:18,883 - mmdet.ssod - INFO - initialize FPN with init_cfg {'type': 'Xavier', 'layer': 'Conv2d', 'distribution': 'uniform'}
2021-11-10 14:11:18,906 - mmdet.ssod - INFO - initialize RPNHead with init_cfg {'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01}
2021-11-10 14:11:18,910 - mmdet.ssod - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'layer': 'Linear', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}]
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
loading annotations into memory...
Done (t=0.03s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
2021-11-10 14:11:24,156 - mmdet.ssod - INFO - Start running, host: lfgp@LAPTOP-VI0T98FT, work_dir: /mnt/c/Users/Francisco Pereira/Desktop/IST/10º semestre/algoritmos/SoftTeacher/work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k
2021-11-10 14:11:24,157 - mmdet.ssod - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH ) StepLrUpdaterHook
(ABOVE_NORMAL) Fp16OptimizerHook
(NORMAL ) CheckpointHook
(NORMAL ) WeightSummary
(NORMAL ) MeanTeacher
(80 ) SubModulesDistEvalHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) WandbLoggerHook
--------------------
before_train_epoch:
(VERY_HIGH ) StepLrUpdaterHook
(NORMAL ) NumClassCheckHook
(LOW ) IterTimerHook
(80 ) SubModulesDistEvalHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) WandbLoggerHook
--------------------
before_train_iter:
(VERY_HIGH ) StepLrUpdaterHook
(NORMAL ) MeanTeacher
(LOW ) IterTimerHook
(80 ) SubModulesDistEvalHook
--------------------
after_train_iter:
(ABOVE_NORMAL) Fp16OptimizerHook
(NORMAL ) CheckpointHook
(NORMAL ) MeanTeacher
(LOW ) IterTimerHook
(80 ) SubModulesDistEvalHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) WandbLoggerHook
--------------------
after_train_epoch:
(NORMAL ) CheckpointHook
(80 ) SubModulesDistEvalHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) WandbLoggerHook
--------------------
before_val_epoch:
(NORMAL ) NumClassCheckHook
(LOW ) IterTimerHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) WandbLoggerHook
--------------------
before_val_iter:
(LOW ) IterTimerHook
--------------------
after_val_iter:
(LOW ) IterTimerHook
--------------------
after_val_epoch:
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) WandbLoggerHook
--------------------
after_run:
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) WandbLoggerHook
--------------------
2021-11-10 14:11:24,160 - mmdet.ssod - INFO - workflow: [('train', 1)], max: 720000 iters
2021-11-10 14:11:24,163 - mmdet.ssod - INFO - Checkpoints will be saved to /mnt/c/Users/Francisco Pereira/Desktop/IST/10º semestre/algoritmos/SoftTeacher/work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k by HardDiskBackend.
2021-11-10 14:11:24,288 - mmdet.ssod - INFO -
+--------------------------------------------------------------------------------------------------------------------+
| Model Information |
+------------------------------------------------+-----------+---------------+-----------------------+------+--------+
| Name | Optimized | Shape | Value Scale [Min,Max] | Lr | Wd |
+------------------------------------------------+-----------+---------------+-----------------------+------+--------+
| teacher.backbone.conv1.weight | N | 64X3X7X7 | Min:-0.671 Max:0.704 | 0.01 | 0.0001 |
| teacher.backbone.bn1.weight | N | 64 | Min:0.513 Max:2.669 | 0.01 | 0.0001 |
| teacher.backbone.bn1.bias | N | 64 | Min:-2.654 Max:6.354 | 0.01 | 0.0001 |
| teacher.backbone.layer1.0.conv1.weight | N | 64X64X1X1 | Min:-0.717 Max:0.392 | 0.01 | 0.0001 |
| teacher.backbone.layer1.0.bn1.weight | N | 64 | Min:0.509 Max:2.066 | 0.01 | 0.0001 |
| teacher.backbone.layer1.0.bn1.bias | N | 64 | Min:-2.411 Max:3.608 | 0.01 | 0.0001 |
| teacher.backbone.layer1.0.conv2.weight | N | 64X64X3X3 | Min:-0.390 Max:0.364 | 0.01 | 0.0001 |
| teacher.backbone.layer1.0.bn2.weight | N | 64 | Min:0.420 Max:2.530 | 0.01 | 0.0001 |
| teacher.backbone.layer1.0.bn2.bias | N | 64 | Min:-2.286 Max:5.913 | 0.01 | 0.0001 |
| teacher.backbone.layer1.0.conv3.weight | N | 256X64X1X1 | Min:-0.397 Max:0.348 | 0.01 | 0.0001 |
| teacher.backbone.layer1.0.bn3.weight | N | 256 | Min:0.011 Max:2.820 | 0.01 | 0.0001 |
| teacher.backbone.layer1.0.bn3.bias | N | 256 | Min:-1.126 Max:1.522 | 0.01 | 0.0001 |
| teacher.backbone.layer1.0.downsample.0.weight | N | 256X64X1X1 | Min:-0.772 Max:0.900 | 0.01 | 0.0001 |
| teacher.backbone.layer1.0.downsample.1.weight | N | 256 | Min:0.004 Max:3.064 | 0.01 | 0.0001 |
| teacher.backbone.layer1.0.downsample.1.bias | N | 256 | Min:-1.126 Max:1.522 | 0.01 | 0.0001 |
| teacher.backbone.layer1.1.conv1.weight | N | 64X256X1X1 | Min:-0.297 Max:0.220 | 0.01 | 0.0001 |
| teacher.backbone.layer1.1.bn1.weight | N | 64 | Min:0.746 Max:1.949 | 0.01 | 0.0001 |
| teacher.backbone.layer1.1.bn1.bias | N | 64 | Min:-1.688 Max:1.578 | 0.01 | 0.0001 |
| teacher.backbone.layer1.1.conv2.weight | N | 64X64X3X3 | Min:-0.240 Max:0.318 | 0.01 | 0.0001 |
| teacher.backbone.layer1.1.bn2.weight | N | 64 | Min:0.621 Max:1.618 | 0.01 | 0.0001 |
| teacher.backbone.layer1.1.bn2.bias | N | 64 | Min:-2.003 Max:2.398 | 0.01 | 0.0001 |
| teacher.backbone.layer1.1.conv3.weight | N | 256X64X1X1 | Min:-0.240 Max:0.280 | 0.01 | 0.0001 |
| teacher.backbone.layer1.1.bn3.weight | N | 256 | Min:-0.017 Max:2.130 | 0.01 | 0.0001 |
| teacher.backbone.layer1.1.bn3.bias | N | 256 | Min:-1.711 Max:1.291 | 0.01 | 0.0001 |
| teacher.backbone.layer1.2.conv1.weight | N | 64X256X1X1 | Min:-0.210 Max:0.264 | 0.01 | 0.0001 |
| teacher.backbone.layer1.2.bn1.weight | N | 64 | Min:0.574 Max:1.688 | 0.01 | 0.0001 |
| teacher.backbone.layer1.2.bn1.bias | N | 64 | Min:-1.876 Max:1.090 | 0.01 | 0.0001 |
| teacher.backbone.layer1.2.conv2.weight | N | 64X64X3X3 | Min:-0.218 Max:0.201 | 0.01 | 0.0001 |
| teacher.backbone.layer1.2.bn2.weight | N | 64 | Min:0.757 Max:1.649 | 0.01 | 0.0001 |
| teacher.backbone.layer1.2.bn2.bias | N | 64 | Min:-2.221 Max:1.878 | 0.01 | 0.0001 |
| teacher.backbone.layer1.2.conv3.weight | N | 256X64X1X1 | Min:-0.275 Max:0.350 | 0.01 | 0.0001 |
| teacher.backbone.layer1.2.bn3.weight | N | 256 | Min:-0.058 Max:2.154 | 0.01 | 0.0001 |
| teacher.backbone.layer1.2.bn3.bias | N | 256 | Min:-1.570 Max:1.535 | 0.01 | 0.0001 |
| teacher.backbone.layer2.0.conv1.weight | N | 128X256X1X1 | Min:-0.334 Max:0.300 | 0.01 | 0.0001 |
| teacher.backbone.layer2.0.bn1.weight | N | 128 | Min:0.610 Max:1.642 | 0.01 | 0.0001 |
| teacher.backbone.layer2.0.bn1.bias | N | 128 | Min:-1.579 Max:1.449 | 0.01 | 0.0001 |
| teacher.backbone.layer2.0.conv2.weight | N | 128X128X3X3 | Min:-0.384 Max:0.377 | 0.01 | 0.0001 |
| teacher.backbone.layer2.0.bn2.weight | N | 128 | Min:0.605 Max:1.622 | 0.01 | 0.0001 |
| teacher.backbone.layer2.0.bn2.bias | N | 128 | Min:-2.768 Max:1.747 | 0.01 | 0.0001 |
| teacher.backbone.layer2.0.conv3.weight | N | 512X128X1X1 | Min:-0.374 Max:0.434 | 0.01 | 0.0001 |
| teacher.backbone.layer2.0.bn3.weight | N | 512 | Min:-0.007 Max:2.730 | 0.01 | 0.0001 |
| teacher.backbone.layer2.0.bn3.bias | N | 512 | Min:-1.545 Max:1.256 | 0.01 | 0.0001 |
| teacher.backbone.layer2.0.downsample.0.weight | N | 512X256X1X1 | Min:-0.466 Max:0.642 | 0.01 | 0.0001 |
| teacher.backbone.layer2.0.downsample.1.weight | N | 512 | Min:0.006 Max:2.552 | 0.01 | 0.0001 |
| teacher.backbone.layer2.0.downsample.1.bias | N | 512 | Min:-1.545 Max:1.256 | 0.01 | 0.0001 |
| teacher.backbone.layer2.1.conv1.weight | N | 128X512X1X1 | Min:-0.162 Max:0.195 | 0.01 | 0.0001 |
| teacher.backbone.layer2.1.bn1.weight | N | 128 | Min:0.578 Max:1.429 | 0.01 | 0.0001 |
| teacher.backbone.layer2.1.bn1.bias | N | 128 | Min:-4.348 Max:0.588 | 0.01 | 0.0001 |
| teacher.backbone.layer2.1.conv2.weight | N | 128X128X3X3 | Min:-0.176 Max:0.177 | 0.01 | 0.0001 |
| teacher.backbone.layer2.1.bn2.weight | N | 128 | Min:0.511 Max:1.794 | 0.01 | 0.0001 |
| teacher.backbone.layer2.1.bn2.bias | N | 128 | Min:-3.825 Max:1.343 | 0.01 | 0.0001 |
| teacher.backbone.layer2.1.conv3.weight | N | 512X128X1X1 | Min:-0.344 Max:0.336 | 0.01 | 0.0001 |
| teacher.backbone.layer2.1.bn3.weight | N | 512 | Min:-0.072 Max:2.122 | 0.01 | 0.0001 |
| teacher.backbone.layer2.1.bn3.bias | N | 512 | Min:-1.502 Max:1.166 | 0.01 | 0.0001 |
| teacher.backbone.layer2.2.conv1.weight | N | 128X512X1X1 | Min:-0.330 Max:0.369 | 0.01 | 0.0001 |
| teacher.backbone.layer2.2.bn1.weight | N | 128 | Min:0.406 Max:1.696 | 0.01 | 0.0001 |
| teacher.backbone.layer2.2.bn1.bias | N | 128 | Min:-2.696 Max:1.944 | 0.01 | 0.0001 |
| teacher.backbone.layer2.2.conv2.weight | N | 128X128X3X3 | Min:-0.326 Max:0.374 | 0.01 | 0.0001 |
| teacher.backbone.layer2.2.bn2.weight | N | 128 | Min:0.460 Max:2.179 | 0.01 | 0.0001 |
| teacher.backbone.layer2.2.bn2.bias | N | 128 | Min:-1.587 Max:0.589 | 0.01 | 0.0001 |
| teacher.backbone.layer2.2.conv3.weight | N | 512X128X1X1 | Min:-0.288 Max:0.232 | 0.01 | 0.0001 |
| teacher.backbone.layer2.2.bn3.weight | N | 512 | Min:-0.006 Max:3.043 | 0.01 | 0.0001 |
| teacher.backbone.layer2.2.bn3.bias | N | 512 | Min:-2.369 Max:0.440 | 0.01 | 0.0001 |
| teacher.backbone.layer2.3.conv1.weight | N | 128X512X1X1 | Min:-0.298 Max:0.346 | 0.01 | 0.0001 |
| teacher.backbone.layer2.3.bn1.weight | N | 128 | Min:0.736 Max:2.394 | 0.01 | 0.0001 |
| teacher.backbone.layer2.3.bn1.bias | N | 128 | Min:-2.643 Max:0.756 | 0.01 | 0.0001 |
| teacher.backbone.layer2.3.conv2.weight | N | 128X128X3X3 | Min:-0.272 Max:0.208 | 0.01 | 0.0001 |
| teacher.backbone.layer2.3.bn2.weight | N | 128 | Min:0.682 Max:1.694 | 0.01 | 0.0001 |
| teacher.backbone.layer2.3.bn2.bias | N | 128 | Min:-1.365 Max:1.599 | 0.01 | 0.0001 |
| teacher.backbone.layer2.3.conv3.weight | N | 512X128X1X1 | Min:-0.279 Max:0.281 | 0.01 | 0.0001 |
| teacher.backbone.layer2.3.bn3.weight | N | 512 | Min:-0.009 Max:1.721 | 0.01 | 0.0001 |
| teacher.backbone.layer2.3.bn3.bias | N | 512 | Min:-1.897 Max:1.182 | 0.01 | 0.0001 |
| teacher.backbone.layer3.0.conv1.weight | N | 256X512X1X1 | Min:-0.230 Max:0.341 | 0.01 | 0.0001 |
| teacher.backbone.layer3.0.bn1.weight | N | 256 | Min:0.621 Max:1.636 | 0.01 | 0.0001 |
| teacher.backbone.layer3.0.bn1.bias | N | 256 | Min:-1.420 Max:0.917 | 0.01 | 0.0001 |
| teacher.backbone.layer3.0.conv2.weight | N | 256X256X3X3 | Min:-0.267 Max:0.179 | 0.01 | 0.0001 |
| teacher.backbone.layer3.0.bn2.weight | N | 256 | Min:0.585 Max:1.749 | 0.01 | 0.0001 |
| teacher.backbone.layer3.0.bn2.bias | N | 256 | Min:-1.837 Max:1.398 | 0.01 | 0.0001 |
| teacher.backbone.layer3.0.conv3.weight | N | 1024X256X1X1 | Min:-0.333 Max:0.384 | 0.01 | 0.0001 |
| teacher.backbone.layer3.0.bn3.weight | N | 1024 | Min:0.071 Max:2.367 | 0.01 | 0.0001 |
| teacher.backbone.layer3.0.bn3.bias | N | 1024 | Min:-0.938 Max:0.887 | 0.01 | 0.0001 |
| teacher.backbone.layer3.0.downsample.0.weight | N | 1024X512X1X1 | Min:-0.333 Max:0.421 | 0.01 | 0.0001 |
| teacher.backbone.layer3.0.downsample.1.weight | N | 1024 | Min:0.034 Max:2.779 | 0.01 | 0.0001 |
| teacher.backbone.layer3.0.downsample.1.bias | N | 1024 | Min:-0.938 Max:0.887 | 0.01 | 0.0001 |
| teacher.backbone.layer3.1.conv1.weight | N | 256X1024X1X1 | Min:-0.197 Max:0.236 | 0.01 | 0.0001 |
| teacher.backbone.layer3.1.bn1.weight | N | 256 | Min:0.566 Max:1.743 | 0.01 | 0.0001 |
| teacher.backbone.layer3.1.bn1.bias | N | 256 | Min:-2.703 Max:1.042 | 0.01 | 0.0001 |
| teacher.backbone.layer3.1.conv2.weight | N | 256X256X3X3 | Min:-0.436 Max:0.196 | 0.01 | 0.0001 |
| teacher.backbone.layer3.1.bn2.weight | N | 256 | Min:0.515 Max:2.301 | 0.01 | 0.0001 |
| teacher.backbone.layer3.1.bn2.bias | N | 256 | Min:-2.548 Max:1.856 | 0.01 | 0.0001 |
| teacher.backbone.layer3.1.conv3.weight | N | 1024X256X1X1 | Min:-0.438 Max:0.295 | 0.01 | 0.0001 |
| teacher.backbone.layer3.1.bn3.weight | N | 1024 | Min:0.055 Max:1.943 | 0.01 | 0.0001 |
| teacher.backbone.layer3.1.bn3.bias | N | 1024 | Min:-1.647 Max:1.016 | 0.01 | 0.0001 |
| teacher.backbone.layer3.2.conv1.weight | N | 256X1024X1X1 | Min:-0.387 Max:0.337 | 0.01 | 0.0001 |
| teacher.backbone.layer3.2.bn1.weight | N | 256 | Min:0.463 Max:1.886 | 0.01 | 0.0001 |
| teacher.backbone.layer3.2.bn1.bias | N | 256 | Min:-2.399 Max:0.488 | 0.01 | 0.0001 |
| teacher.backbone.layer3.2.conv2.weight | N | 256X256X3X3 | Min:-0.165 Max:0.258 | 0.01 | 0.0001 |
| teacher.backbone.layer3.2.bn2.weight | N | 256 | Min:0.555 Max:1.901 | 0.01 | 0.0001 |
| teacher.backbone.layer3.2.bn2.bias | N | 256 | Min:-1.655 Max:0.704 | 0.01 | 0.0001 |
| teacher.backbone.layer3.2.conv3.weight | N | 1024X256X1X1 | Min:-0.290 Max:0.261 | 0.01 | 0.0001 |
| teacher.backbone.layer3.2.bn3.weight | N | 1024 | Min:0.049 Max:1.450 | 0.01 | 0.0001 |
| teacher.backbone.layer3.2.bn3.bias | N | 1024 | Min:-1.201 Max:0.587 | 0.01 | 0.0001 |
| teacher.backbone.layer3.3.conv1.weight | N | 256X1024X1X1 | Min:-0.194 Max:0.295 | 0.01 | 0.0001 |
| teacher.backbone.layer3.3.bn1.weight | N | 256 | Min:0.442 Max:1.353 | 0.01 | 0.0001 |
| teacher.backbone.layer3.3.bn1.bias | N | 256 | Min:-2.322 Max:0.509 | 0.01 | 0.0001 |
| teacher.backbone.layer3.3.conv2.weight | N | 256X256X3X3 | Min:-0.201 Max:0.176 | 0.01 | 0.0001 |
| teacher.backbone.layer3.3.bn2.weight | N | 256 | Min:0.529 Max:1.939 | 0.01 | 0.0001 |
| teacher.backbone.layer3.3.bn2.bias | N | 256 | Min:-1.610 Max:0.776 | 0.01 | 0.0001 |
| teacher.backbone.layer3.3.conv3.weight | N | 1024X256X1X1 | Min:-0.205 Max:0.239 | 0.01 | 0.0001 |
| teacher.backbone.layer3.3.bn3.weight | N | 1024 | Min:-0.037 Max:1.646 | 0.01 | 0.0001 |
| teacher.backbone.layer3.3.bn3.bias | N | 1024 | Min:-1.484 Max:0.344 | 0.01 | 0.0001 |
| teacher.backbone.layer3.4.conv1.weight | N | 256X1024X1X1 | Min:-0.226 Max:0.306 | 0.01 | 0.0001 |
| teacher.backbone.layer3.4.bn1.weight | N | 256 | Min:0.438 Max:1.446 | 0.01 | 0.0001 |
| teacher.backbone.layer3.4.bn1.bias | N | 256 | Min:-2.511 Max:0.557 | 0.01 | 0.0001 |
| teacher.backbone.layer3.4.conv2.weight | N | 256X256X3X3 | Min:-0.147 Max:0.223 | 0.01 | 0.0001 |
| teacher.backbone.layer3.4.bn2.weight | N | 256 | Min:0.651 Max:1.858 | 0.01 | 0.0001 |
| teacher.backbone.layer3.4.bn2.bias | N | 256 | Min:-1.588 Max:0.661 | 0.01 | 0.0001 |
| teacher.backbone.layer3.4.conv3.weight | N | 1024X256X1X1 | Min:-0.178 Max:0.265 | 0.01 | 0.0001 |
| teacher.backbone.layer3.4.bn3.weight | N | 1024 | Min:-0.001 Max:1.501 | 0.01 | 0.0001 |
| teacher.backbone.layer3.4.bn3.bias | N | 1024 | Min:-1.108 Max:0.639 | 0.01 | 0.0001 |
| teacher.backbone.layer3.5.conv1.weight | N | 256X1024X1X1 | Min:-0.153 Max:0.330 | 0.01 | 0.0001 |
| teacher.backbone.layer3.5.bn1.weight | N | 256 | Min:0.425 Max:1.547 | 0.01 | 0.0001 |
| teacher.backbone.layer3.5.bn1.bias | N | 256 | Min:-1.972 Max:0.823 | 0.01 | 0.0001 |
| teacher.backbone.layer3.5.conv2.weight | N | 256X256X3X3 | Min:-0.293 Max:0.276 | 0.01 | 0.0001 |
| teacher.backbone.layer3.5.bn2.weight | N | 256 | Min:0.650 Max:2.942 | 0.01 | 0.0001 |
| teacher.backbone.layer3.5.bn2.bias | N | 256 | Min:-1.093 Max:0.771 | 0.01 | 0.0001 |
| teacher.backbone.layer3.5.conv3.weight | N | 1024X256X1X1 | Min:-0.232 Max:0.294 | 0.01 | 0.0001 |
| teacher.backbone.layer3.5.bn3.weight | N | 1024 | Min:0.004 Max:1.984 | 0.01 | 0.0001 |
| teacher.backbone.layer3.5.bn3.bias | N | 1024 | Min:-1.636 Max:1.250 | 0.01 | 0.0001 |
| teacher.backbone.layer4.0.conv1.weight | N | 512X1024X1X1 | Min:-0.184 Max:0.331 | 0.01 | 0.0001 |
| teacher.backbone.layer4.0.bn1.weight | N | 512 | Min:0.535 Max:1.594 | 0.01 | 0.0001 |
| teacher.backbone.layer4.0.bn1.bias | N | 512 | Min:-1.756 Max:0.288 | 0.01 | 0.0001 |
| teacher.backbone.layer4.0.conv2.weight | N | 512X512X3X3 | Min:-0.175 Max:0.272 | 0.01 | 0.0001 |
| teacher.backbone.layer4.0.bn2.weight | N | 512 | Min:0.456 Max:1.542 | 0.01 | 0.0001 |
| teacher.backbone.layer4.0.bn2.bias | N | 512 | Min:-1.820 Max:0.839 | 0.01 | 0.0001 |
| teacher.backbone.layer4.0.conv3.weight | N | 2048X512X1X1 | Min:-0.332 Max:0.432 | 0.01 | 0.0001 |
| teacher.backbone.layer4.0.bn3.weight | N | 2048 | Min:0.888 Max:3.492 | 0.01 | 0.0001 |
| teacher.backbone.layer4.0.bn3.bias | N | 2048 | Min:-1.810 Max:0.980 | 0.01 | 0.0001 |
| teacher.backbone.layer4.0.downsample.0.weight | N | 2048X1024X1X1 | Min:-0.622 Max:0.465 | 0.01 | 0.0001 |
| teacher.backbone.layer4.0.downsample.1.weight | N | 2048 | Min:0.261 Max:4.575 | 0.01 | 0.0001 |
| teacher.backbone.layer4.0.downsample.1.bias | N | 2048 | Min:-1.810 Max:0.980 | 0.01 | 0.0001 |
| teacher.backbone.layer4.1.conv1.weight | N | 512X2048X1X1 | Min:-0.316 Max:0.577 | 0.01 | 0.0001 |
| teacher.backbone.layer4.1.bn1.weight | N | 512 | Min:0.398 Max:1.429 | 0.01 | 0.0001 |
| teacher.backbone.layer4.1.bn1.bias | N | 512 | Min:-1.380 Max:0.428 | 0.01 | 0.0001 |
| teacher.backbone.layer4.1.conv2.weight | N | 512X512X3X3 | Min:-0.217 Max:0.284 | 0.01 | 0.0001 |
| teacher.backbone.layer4.1.bn2.weight | N | 512 | Min:0.349 Max:1.550 | 0.01 | 0.0001 |
| teacher.backbone.layer4.1.bn2.bias | N | 512 | Min:-1.867 Max:0.880 | 0.01 | 0.0001 |
| teacher.backbone.layer4.1.conv3.weight | N | 2048X512X1X1 | Min:-0.200 Max:0.277 | 0.01 | 0.0001 |
| teacher.backbone.layer4.1.bn3.weight | N | 2048 | Min:0.574 Max:2.847 | 0.01 | 0.0001 |
| teacher.backbone.layer4.1.bn3.bias | N | 2048 | Min:-2.638 Max:0.544 | 0.01 | 0.0001 |
| teacher.backbone.layer4.2.conv1.weight | N | 512X2048X1X1 | Min:-0.289 Max:0.514 | 0.01 | 0.0001 |
| teacher.backbone.layer4.2.bn1.weight | N | 512 | Min:0.366 Max:1.249 | 0.01 | 0.0001 |
| teacher.backbone.layer4.2.bn1.bias | N | 512 | Min:-1.664 Max:0.753 | 0.01 | 0.0001 |
| teacher.backbone.layer4.2.conv2.weight | N | 512X512X3X3 | Min:-0.142 Max:0.144 | 0.01 | 0.0001 |
| teacher.backbone.layer4.2.bn2.weight | N | 512 | Min:0.516 Max:1.335 | 0.01 | 0.0001 |
| teacher.backbone.layer4.2.bn2.bias | N | 512 | Min:-1.871 Max:1.181 | 0.01 | 0.0001 |
| teacher.backbone.layer4.2.conv3.weight | N | 2048X512X1X1 | Min:-0.135 Max:0.300 | 0.01 | 0.0001 |
| teacher.backbone.layer4.2.bn3.weight | N | 2048 | Min:0.435 Max:3.073 | 0.01 | 0.0001 |
| teacher.backbone.layer4.2.bn3.bias | N | 2048 | Min:-3.885 Max:-0.249 | 0.01 | 0.0001 |
| teacher.neck.lateral_convs.0.conv.weight | N | 256X256X1X1 | Min:-0.108 Max:0.108 | 0.01 | 0.0001 |
| teacher.neck.lateral_convs.0.conv.bias | N | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| teacher.neck.lateral_convs.1.conv.weight | N | 256X512X1X1 | Min:-0.088 Max:0.088 | 0.01 | 0.0001 |
| teacher.neck.lateral_convs.1.conv.bias | N | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| teacher.neck.lateral_convs.2.conv.weight | N | 256X1024X1X1 | Min:-0.068 Max:0.068 | 0.01 | 0.0001 |
| teacher.neck.lateral_convs.2.conv.bias | N | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| teacher.neck.lateral_convs.3.conv.weight | N | 256X2048X1X1 | Min:-0.051 Max:0.051 | 0.01 | 0.0001 |
| teacher.neck.lateral_convs.3.conv.bias | N | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| teacher.neck.fpn_convs.0.conv.weight | N | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.01 | 0.0001 |
| teacher.neck.fpn_convs.0.conv.bias | N | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| teacher.neck.fpn_convs.1.conv.weight | N | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.01 | 0.0001 |
| teacher.neck.fpn_convs.1.conv.bias | N | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| teacher.neck.fpn_convs.2.conv.weight | N | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.01 | 0.0001 |
| teacher.neck.fpn_convs.2.conv.bias | N | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| teacher.neck.fpn_convs.3.conv.weight | N | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.01 | 0.0001 |
| teacher.neck.fpn_convs.3.conv.bias | N | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| teacher.rpn_head.rpn_conv.weight | N | 256X256X3X3 | Min:-0.045 Max:0.049 | 0.01 | 0.0001 |
| teacher.rpn_head.rpn_conv.bias | N | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| teacher.rpn_head.rpn_cls.weight | N | 3X256X1X1 | Min:-0.029 Max:0.038 | 0.01 | 0.0001 |
| teacher.rpn_head.rpn_cls.bias | N | 3 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| teacher.rpn_head.rpn_reg.weight | N | 12X256X1X1 | Min:-0.033 Max:0.036 | 0.01 | 0.0001 |
| teacher.rpn_head.rpn_reg.bias | N | 12 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| teacher.roi_head.bbox_head.fc_cls.weight | N | 81X1024 | Min:-0.178 Max:0.187 | 0.01 | 0.0001 |
| teacher.roi_head.bbox_head.fc_cls.bias | N | 81 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| teacher.roi_head.bbox_head.fc_reg.weight | N | 320X1024 | Min:-0.188 Max:0.188 | 0.01 | 0.0001 |
| teacher.roi_head.bbox_head.fc_reg.bias | N | 320 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| teacher.roi_head.bbox_head.shared_fcs.0.weight | N | 1024X12544 | Min:-0.064 Max:0.063 | 0.01 | 0.0001 |
| teacher.roi_head.bbox_head.shared_fcs.0.bias | N | 1024 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| teacher.roi_head.bbox_head.shared_fcs.1.weight | N | 1024X1024 | Min:-0.150 Max:0.163 | 0.01 | 0.0001 |
| teacher.roi_head.bbox_head.shared_fcs.1.bias | N | 1024 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.backbone.conv1.weight | N | 64X3X7X7 | Min:-0.671 Max:0.704 | 0.01 | 0.0001 |
| student.backbone.bn1.weight | N | 64 | Min:0.513 Max:2.669 | 0.01 | 0.0001 |
| student.backbone.bn1.bias | N | 64 | Min:-2.654 Max:6.354 | 0.01 | 0.0001 |
| student.backbone.layer1.0.conv1.weight | N | 64X64X1X1 | Min:-0.717 Max:0.392 | 0.01 | 0.0001 |
| student.backbone.layer1.0.bn1.weight | N | 64 | Min:0.509 Max:2.066 | 0.01 | 0.0001 |
| student.backbone.layer1.0.bn1.bias | N | 64 | Min:-2.411 Max:3.608 | 0.01 | 0.0001 |
| student.backbone.layer1.0.conv2.weight | N | 64X64X3X3 | Min:-0.390 Max:0.364 | 0.01 | 0.0001 |
| student.backbone.layer1.0.bn2.weight | N | 64 | Min:0.420 Max:2.530 | 0.01 | 0.0001 |
| student.backbone.layer1.0.bn2.bias | N | 64 | Min:-2.286 Max:5.913 | 0.01 | 0.0001 |
| student.backbone.layer1.0.conv3.weight | N | 256X64X1X1 | Min:-0.397 Max:0.348 | 0.01 | 0.0001 |
| student.backbone.layer1.0.bn3.weight | N | 256 | Min:0.011 Max:2.820 | 0.01 | 0.0001 |
| student.backbone.layer1.0.bn3.bias | N | 256 | Min:-1.126 Max:1.522 | 0.01 | 0.0001 |
| student.backbone.layer1.0.downsample.0.weight | N | 256X64X1X1 | Min:-0.772 Max:0.900 | 0.01 | 0.0001 |
| student.backbone.layer1.0.downsample.1.weight | N | 256 | Min:0.004 Max:3.064 | 0.01 | 0.0001 |
| student.backbone.layer1.0.downsample.1.bias | N | 256 | Min:-1.126 Max:1.522 | 0.01 | 0.0001 |
| student.backbone.layer1.1.conv1.weight | N | 64X256X1X1 | Min:-0.297 Max:0.220 | 0.01 | 0.0001 |
| student.backbone.layer1.1.bn1.weight | N | 64 | Min:0.746 Max:1.949 | 0.01 | 0.0001 |
| student.backbone.layer1.1.bn1.bias | N | 64 | Min:-1.688 Max:1.578 | 0.01 | 0.0001 |
| student.backbone.layer1.1.conv2.weight | N | 64X64X3X3 | Min:-0.240 Max:0.318 | 0.01 | 0.0001 |
| student.backbone.layer1.1.bn2.weight | N | 64 | Min:0.621 Max:1.618 | 0.01 | 0.0001 |
| student.backbone.layer1.1.bn2.bias | N | 64 | Min:-2.003 Max:2.398 | 0.01 | 0.0001 |
| student.backbone.layer1.1.conv3.weight | N | 256X64X1X1 | Min:-0.240 Max:0.280 | 0.01 | 0.0001 |
| student.backbone.layer1.1.bn3.weight | N | 256 | Min:-0.017 Max:2.130 | 0.01 | 0.0001 |
| student.backbone.layer1.1.bn3.bias | N | 256 | Min:-1.711 Max:1.291 | 0.01 | 0.0001 |
| student.backbone.layer1.2.conv1.weight | N | 64X256X1X1 | Min:-0.210 Max:0.264 | 0.01 | 0.0001 |
| student.backbone.layer1.2.bn1.weight | N | 64 | Min:0.574 Max:1.688 | 0.01 | 0.0001 |
| student.backbone.layer1.2.bn1.bias | N | 64 | Min:-1.876 Max:1.090 | 0.01 | 0.0001 |
| student.backbone.layer1.2.conv2.weight | N | 64X64X3X3 | Min:-0.218 Max:0.201 | 0.01 | 0.0001 |
| student.backbone.layer1.2.bn2.weight | N | 64 | Min:0.757 Max:1.649 | 0.01 | 0.0001 |
| student.backbone.layer1.2.bn2.bias | N | 64 | Min:-2.221 Max:1.878 | 0.01 | 0.0001 |
| student.backbone.layer1.2.conv3.weight | N | 256X64X1X1 | Min:-0.275 Max:0.350 | 0.01 | 0.0001 |
| student.backbone.layer1.2.bn3.weight | N | 256 | Min:-0.058 Max:2.154 | 0.01 | 0.0001 |
| student.backbone.layer1.2.bn3.bias | N | 256 | Min:-1.570 Max:1.535 | 0.01 | 0.0001 |
| student.backbone.layer2.0.conv1.weight | Y | 128X256X1X1 | Min:-0.334 Max:0.300 | 0.01 | 0.0001 |
| student.backbone.layer2.0.bn1.weight | N | 128 | Min:0.610 Max:1.642 | 0.01 | 0.0001 |
| student.backbone.layer2.0.bn1.bias | N | 128 | Min:-1.579 Max:1.449 | 0.01 | 0.0001 |
| student.backbone.layer2.0.conv2.weight | Y | 128X128X3X3 | Min:-0.384 Max:0.377 | 0.01 | 0.0001 |
| student.backbone.layer2.0.bn2.weight | N | 128 | Min:0.605 Max:1.622 | 0.01 | 0.0001 |
| student.backbone.layer2.0.bn2.bias | N | 128 | Min:-2.768 Max:1.747 | 0.01 | 0.0001 |
| student.backbone.layer2.0.conv3.weight | Y | 512X128X1X1 | Min:-0.374 Max:0.434 | 0.01 | 0.0001 |
| student.backbone.layer2.0.bn3.weight | N | 512 | Min:-0.007 Max:2.730 | 0.01 | 0.0001 |
| student.backbone.layer2.0.bn3.bias | N | 512 | Min:-1.545 Max:1.256 | 0.01 | 0.0001 |
| student.backbone.layer2.0.downsample.0.weight | Y | 512X256X1X1 | Min:-0.466 Max:0.642 | 0.01 | 0.0001 |
| student.backbone.layer2.0.downsample.1.weight | N | 512 | Min:0.006 Max:2.552 | 0.01 | 0.0001 |
| student.backbone.layer2.0.downsample.1.bias | N | 512 | Min:-1.545 Max:1.256 | 0.01 | 0.0001 |
| student.backbone.layer2.1.conv1.weight | Y | 128X512X1X1 | Min:-0.162 Max:0.195 | 0.01 | 0.0001 |
| student.backbone.layer2.1.bn1.weight | N | 128 | Min:0.578 Max:1.429 | 0.01 | 0.0001 |
| student.backbone.layer2.1.bn1.bias | N | 128 | Min:-4.348 Max:0.588 | 0.01 | 0.0001 |
| student.backbone.layer2.1.conv2.weight | Y | 128X128X3X3 | Min:-0.176 Max:0.177 | 0.01 | 0.0001 |
| student.backbone.layer2.1.bn2.weight | N | 128 | Min:0.511 Max:1.794 | 0.01 | 0.0001 |
| student.backbone.layer2.1.bn2.bias | N | 128 | Min:-3.825 Max:1.343 | 0.01 | 0.0001 |
| student.backbone.layer2.1.conv3.weight | Y | 512X128X1X1 | Min:-0.344 Max:0.336 | 0.01 | 0.0001 |
| student.backbone.layer2.1.bn3.weight | N | 512 | Min:-0.072 Max:2.122 | 0.01 | 0.0001 |
| student.backbone.layer2.1.bn3.bias | N | 512 | Min:-1.502 Max:1.166 | 0.01 | 0.0001 |
| student.backbone.layer2.2.conv1.weight | Y | 128X512X1X1 | Min:-0.330 Max:0.369 | 0.01 | 0.0001 |
| student.backbone.layer2.2.bn1.weight | N | 128 | Min:0.406 Max:1.696 | 0.01 | 0.0001 |
| student.backbone.layer2.2.bn1.bias | N | 128 | Min:-2.696 Max:1.944 | 0.01 | 0.0001 |
| student.backbone.layer2.2.conv2.weight | Y | 128X128X3X3 | Min:-0.326 Max:0.374 | 0.01 | 0.0001 |
| student.backbone.layer2.2.bn2.weight | N | 128 | Min:0.460 Max:2.179 | 0.01 | 0.0001 |
| student.backbone.layer2.2.bn2.bias | N | 128 | Min:-1.587 Max:0.589 | 0.01 | 0.0001 |
| student.backbone.layer2.2.conv3.weight | Y | 512X128X1X1 | Min:-0.288 Max:0.232 | 0.01 | 0.0001 |
| student.backbone.layer2.2.bn3.weight | N | 512 | Min:-0.006 Max:3.043 | 0.01 | 0.0001 |
| student.backbone.layer2.2.bn3.bias | N | 512 | Min:-2.369 Max:0.440 | 0.01 | 0.0001 |
| student.backbone.layer2.3.conv1.weight | Y | 128X512X1X1 | Min:-0.298 Max:0.346 | 0.01 | 0.0001 |
| student.backbone.layer2.3.bn1.weight | N | 128 | Min:0.736 Max:2.394 | 0.01 | 0.0001 |
| student.backbone.layer2.3.bn1.bias | N | 128 | Min:-2.643 Max:0.756 | 0.01 | 0.0001 |
| student.backbone.layer2.3.conv2.weight | Y | 128X128X3X3 | Min:-0.272 Max:0.208 | 0.01 | 0.0001 |
| student.backbone.layer2.3.bn2.weight | N | 128 | Min:0.682 Max:1.694 | 0.01 | 0.0001 |
| student.backbone.layer2.3.bn2.bias | N | 128 | Min:-1.365 Max:1.599 | 0.01 | 0.0001 |
| student.backbone.layer2.3.conv3.weight | Y | 512X128X1X1 | Min:-0.279 Max:0.281 | 0.01 | 0.0001 |
| student.backbone.layer2.3.bn3.weight | N | 512 | Min:-0.009 Max:1.721 | 0.01 | 0.0001 |
| student.backbone.layer2.3.bn3.bias | N | 512 | Min:-1.897 Max:1.182 | 0.01 | 0.0001 |
| student.backbone.layer3.0.conv1.weight | Y | 256X512X1X1 | Min:-0.230 Max:0.341 | 0.01 | 0.0001 |
| student.backbone.layer3.0.bn1.weight | N | 256 | Min:0.621 Max:1.636 | 0.01 | 0.0001 |
| student.backbone.layer3.0.bn1.bias | N | 256 | Min:-1.420 Max:0.917 | 0.01 | 0.0001 |
| student.backbone.layer3.0.conv2.weight | Y | 256X256X3X3 | Min:-0.267 Max:0.179 | 0.01 | 0.0001 |
| student.backbone.layer3.0.bn2.weight | N | 256 | Min:0.585 Max:1.749 | 0.01 | 0.0001 |
| student.backbone.layer3.0.bn2.bias | N | 256 | Min:-1.837 Max:1.398 | 0.01 | 0.0001 |
| student.backbone.layer3.0.conv3.weight | Y | 1024X256X1X1 | Min:-0.333 Max:0.384 | 0.01 | 0.0001 |
| student.backbone.layer3.0.bn3.weight | N | 1024 | Min:0.071 Max:2.367 | 0.01 | 0.0001 |
| student.backbone.layer3.0.bn3.bias | N | 1024 | Min:-0.938 Max:0.887 | 0.01 | 0.0001 |
| student.backbone.layer3.0.downsample.0.weight | Y | 1024X512X1X1 | Min:-0.333 Max:0.421 | 0.01 | 0.0001 |
| student.backbone.layer3.0.downsample.1.weight | N | 1024 | Min:0.034 Max:2.779 | 0.01 | 0.0001 |
| student.backbone.layer3.0.downsample.1.bias | N | 1024 | Min:-0.938 Max:0.887 | 0.01 | 0.0001 |
| student.backbone.layer3.1.conv1.weight | Y | 256X1024X1X1 | Min:-0.197 Max:0.236 | 0.01 | 0.0001 |
| student.backbone.layer3.1.bn1.weight | N | 256 | Min:0.566 Max:1.743 | 0.01 | 0.0001 |
| student.backbone.layer3.1.bn1.bias | N | 256 | Min:-2.703 Max:1.042 | 0.01 | 0.0001 |
| student.backbone.layer3.1.conv2.weight | Y | 256X256X3X3 | Min:-0.436 Max:0.196 | 0.01 | 0.0001 |
| student.backbone.layer3.1.bn2.weight | N | 256 | Min:0.515 Max:2.301 | 0.01 | 0.0001 |
| student.backbone.layer3.1.bn2.bias | N | 256 | Min:-2.548 Max:1.856 | 0.01 | 0.0001 |
| student.backbone.layer3.1.conv3.weight | Y | 1024X256X1X1 | Min:-0.438 Max:0.295 | 0.01 | 0.0001 |
| student.backbone.layer3.1.bn3.weight | N | 1024 | Min:0.055 Max:1.943 | 0.01 | 0.0001 |
| student.backbone.layer3.1.bn3.bias | N | 1024 | Min:-1.647 Max:1.016 | 0.01 | 0.0001 |
| student.backbone.layer3.2.conv1.weight | Y | 256X1024X1X1 | Min:-0.387 Max:0.337 | 0.01 | 0.0001 |
| student.backbone.layer3.2.bn1.weight | N | 256 | Min:0.463 Max:1.886 | 0.01 | 0.0001 |
| student.backbone.layer3.2.bn1.bias | N | 256 | Min:-2.399 Max:0.488 | 0.01 | 0.0001 |
| student.backbone.layer3.2.conv2.weight | Y | 256X256X3X3 | Min:-0.165 Max:0.258 | 0.01 | 0.0001 |
| student.backbone.layer3.2.bn2.weight | N | 256 | Min:0.555 Max:1.901 | 0.01 | 0.0001 |
| student.backbone.layer3.2.bn2.bias | N | 256 | Min:-1.655 Max:0.704 | 0.01 | 0.0001 |
| student.backbone.layer3.2.conv3.weight | Y | 1024X256X1X1 | Min:-0.290 Max:0.261 | 0.01 | 0.0001 |
| student.backbone.layer3.2.bn3.weight | N | 1024 | Min:0.049 Max:1.450 | 0.01 | 0.0001 |
| student.backbone.layer3.2.bn3.bias | N | 1024 | Min:-1.201 Max:0.587 | 0.01 | 0.0001 |
| student.backbone.layer3.3.conv1.weight | Y | 256X1024X1X1 | Min:-0.194 Max:0.295 | 0.01 | 0.0001 |
| student.backbone.layer3.3.bn1.weight | N | 256 | Min:0.442 Max:1.353 | 0.01 | 0.0001 |
| student.backbone.layer3.3.bn1.bias | N | 256 | Min:-2.322 Max:0.509 | 0.01 | 0.0001 |
| student.backbone.layer3.3.conv2.weight | Y | 256X256X3X3 | Min:-0.201 Max:0.176 | 0.01 | 0.0001 |
| student.backbone.layer3.3.bn2.weight | N | 256 | Min:0.529 Max:1.939 | 0.01 | 0.0001 |
| student.backbone.layer3.3.bn2.bias | N | 256 | Min:-1.610 Max:0.776 | 0.01 | 0.0001 |
| student.backbone.layer3.3.conv3.weight | Y | 1024X256X1X1 | Min:-0.205 Max:0.239 | 0.01 | 0.0001 |
| student.backbone.layer3.3.bn3.weight | N | 1024 | Min:-0.037 Max:1.646 | 0.01 | 0.0001 |
| student.backbone.layer3.3.bn3.bias | N | 1024 | Min:-1.484 Max:0.344 | 0.01 | 0.0001 |
| student.backbone.layer3.4.conv1.weight | Y | 256X1024X1X1 | Min:-0.226 Max:0.306 | 0.01 | 0.0001 |
| student.backbone.layer3.4.bn1.weight | N | 256 | Min:0.438 Max:1.446 | 0.01 | 0.0001 |
| student.backbone.layer3.4.bn1.bias | N | 256 | Min:-2.511 Max:0.557 | 0.01 | 0.0001 |
| student.backbone.layer3.4.conv2.weight | Y | 256X256X3X3 | Min:-0.147 Max:0.223 | 0.01 | 0.0001 |
| student.backbone.layer3.4.bn2.weight | N | 256 | Min:0.651 Max:1.858 | 0.01 | 0.0001 |
| student.backbone.layer3.4.bn2.bias | N | 256 | Min:-1.588 Max:0.661 | 0.01 | 0.0001 |
| student.backbone.layer3.4.conv3.weight | Y | 1024X256X1X1 | Min:-0.178 Max:0.265 | 0.01 | 0.0001 |
| student.backbone.layer3.4.bn3.weight | N | 1024 | Min:-0.001 Max:1.501 | 0.01 | 0.0001 |
| student.backbone.layer3.4.bn3.bias | N | 1024 | Min:-1.108 Max:0.639 | 0.01 | 0.0001 |
| student.backbone.layer3.5.conv1.weight | Y | 256X1024X1X1 | Min:-0.153 Max:0.330 | 0.01 | 0.0001 |
| student.backbone.layer3.5.bn1.weight | N | 256 | Min:0.425 Max:1.547 | 0.01 | 0.0001 |
| student.backbone.layer3.5.bn1.bias | N | 256 | Min:-1.972 Max:0.823 | 0.01 | 0.0001 |
| student.backbone.layer3.5.conv2.weight | Y | 256X256X3X3 | Min:-0.293 Max:0.276 | 0.01 | 0.0001 |
| student.backbone.layer3.5.bn2.weight | N | 256 | Min:0.650 Max:2.942 | 0.01 | 0.0001 |
| student.backbone.layer3.5.bn2.bias | N | 256 | Min:-1.093 Max:0.771 | 0.01 | 0.0001 |
| student.backbone.layer3.5.conv3.weight | Y | 1024X256X1X1 | Min:-0.232 Max:0.294 | 0.01 | 0.0001 |
| student.backbone.layer3.5.bn3.weight | N | 1024 | Min:0.004 Max:1.984 | 0.01 | 0.0001 |
| student.backbone.layer3.5.bn3.bias | N | 1024 | Min:-1.636 Max:1.250 | 0.01 | 0.0001 |
| student.backbone.layer4.0.conv1.weight | Y | 512X1024X1X1 | Min:-0.184 Max:0.331 | 0.01 | 0.0001 |
| student.backbone.layer4.0.bn1.weight | N | 512 | Min:0.535 Max:1.594 | 0.01 | 0.0001 |
| student.backbone.layer4.0.bn1.bias | N | 512 | Min:-1.756 Max:0.288 | 0.01 | 0.0001 |
| student.backbone.layer4.0.conv2.weight | Y | 512X512X3X3 | Min:-0.175 Max:0.272 | 0.01 | 0.0001 |
| student.backbone.layer4.0.bn2.weight | N | 512 | Min:0.456 Max:1.542 | 0.01 | 0.0001 |
| student.backbone.layer4.0.bn2.bias | N | 512 | Min:-1.820 Max:0.839 | 0.01 | 0.0001 |
| student.backbone.layer4.0.conv3.weight | Y | 2048X512X1X1 | Min:-0.332 Max:0.432 | 0.01 | 0.0001 |
| student.backbone.layer4.0.bn3.weight | N | 2048 | Min:0.888 Max:3.492 | 0.01 | 0.0001 |
| student.backbone.layer4.0.bn3.bias | N | 2048 | Min:-1.810 Max:0.980 | 0.01 | 0.0001 |
| student.backbone.layer4.0.downsample.0.weight | Y | 2048X1024X1X1 | Min:-0.622 Max:0.465 | 0.01 | 0.0001 |
| student.backbone.layer4.0.downsample.1.weight | N | 2048 | Min:0.261 Max:4.575 | 0.01 | 0.0001 |
| student.backbone.layer4.0.downsample.1.bias | N | 2048 | Min:-1.810 Max:0.980 | 0.01 | 0.0001 |
| student.backbone.layer4.1.conv1.weight | Y | 512X2048X1X1 | Min:-0.316 Max:0.577 | 0.01 | 0.0001 |
| student.backbone.layer4.1.bn1.weight | N | 512 | Min:0.398 Max:1.429 | 0.01 | 0.0001 |
| student.backbone.layer4.1.bn1.bias | N | 512 | Min:-1.380 Max:0.428 | 0.01 | 0.0001 |
| student.backbone.layer4.1.conv2.weight | Y | 512X512X3X3 | Min:-0.217 Max:0.284 | 0.01 | 0.0001 |
| student.backbone.layer4.1.bn2.weight | N | 512 | Min:0.349 Max:1.550 | 0.01 | 0.0001 |
| student.backbone.layer4.1.bn2.bias | N | 512 | Min:-1.867 Max:0.880 | 0.01 | 0.0001 |
| student.backbone.layer4.1.conv3.weight | Y | 2048X512X1X1 | Min:-0.200 Max:0.277 | 0.01 | 0.0001 |
| student.backbone.layer4.1.bn3.weight | N | 2048 | Min:0.574 Max:2.847 | 0.01 | 0.0001 |
| student.backbone.layer4.1.bn3.bias | N | 2048 | Min:-2.638 Max:0.544 | 0.01 | 0.0001 |
| student.backbone.layer4.2.conv1.weight | Y | 512X2048X1X1 | Min:-0.289 Max:0.514 | 0.01 | 0.0001 |
| student.backbone.layer4.2.bn1.weight | N | 512 | Min:0.366 Max:1.249 | 0.01 | 0.0001 |
| student.backbone.layer4.2.bn1.bias | N | 512 | Min:-1.664 Max:0.753 | 0.01 | 0.0001 |
| student.backbone.layer4.2.conv2.weight | Y | 512X512X3X3 | Min:-0.142 Max:0.144 | 0.01 | 0.0001 |
| student.backbone.layer4.2.bn2.weight | N | 512 | Min:0.516 Max:1.335 | 0.01 | 0.0001 |
| student.backbone.layer4.2.bn2.bias | N | 512 | Min:-1.871 Max:1.181 | 0.01 | 0.0001 |
| student.backbone.layer4.2.conv3.weight | Y | 2048X512X1X1 | Min:-0.135 Max:0.300 | 0.01 | 0.0001 |
| student.backbone.layer4.2.bn3.weight | N | 2048 | Min:0.435 Max:3.073 | 0.01 | 0.0001 |
| student.backbone.layer4.2.bn3.bias | N | 2048 | Min:-3.885 Max:-0.249 | 0.01 | 0.0001 |
| student.neck.lateral_convs.0.conv.weight | Y | 256X256X1X1 | Min:-0.108 Max:0.108 | 0.01 | 0.0001 |
| student.neck.lateral_convs.0.conv.bias | Y | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.neck.lateral_convs.1.conv.weight | Y | 256X512X1X1 | Min:-0.088 Max:0.088 | 0.01 | 0.0001 |
| student.neck.lateral_convs.1.conv.bias | Y | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.neck.lateral_convs.2.conv.weight | Y | 256X1024X1X1 | Min:-0.068 Max:0.068 | 0.01 | 0.0001 |
| student.neck.lateral_convs.2.conv.bias | Y | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.neck.lateral_convs.3.conv.weight | Y | 256X2048X1X1 | Min:-0.051 Max:0.051 | 0.01 | 0.0001 |
| student.neck.lateral_convs.3.conv.bias | Y | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.neck.fpn_convs.0.conv.weight | Y | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.01 | 0.0001 |
| student.neck.fpn_convs.0.conv.bias | Y | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.neck.fpn_convs.1.conv.weight | Y | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.01 | 0.0001 |
| student.neck.fpn_convs.1.conv.bias | Y | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.neck.fpn_convs.2.conv.weight | Y | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.01 | 0.0001 |
| student.neck.fpn_convs.2.conv.bias | Y | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.neck.fpn_convs.3.conv.weight | Y | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.01 | 0.0001 |
| student.neck.fpn_convs.3.conv.bias | Y | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.rpn_head.rpn_conv.weight | Y | 256X256X3X3 | Min:-0.045 Max:0.047 | 0.01 | 0.0001 |
| student.rpn_head.rpn_conv.bias | Y | 256 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.rpn_head.rpn_cls.weight | Y | 3X256X1X1 | Min:-0.031 Max:0.034 | 0.01 | 0.0001 |
| student.rpn_head.rpn_cls.bias | Y | 3 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.rpn_head.rpn_reg.weight | Y | 12X256X1X1 | Min:-0.033 Max:0.032 | 0.01 | 0.0001 |
| student.rpn_head.rpn_reg.bias | Y | 12 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.roi_head.bbox_head.fc_cls.weight | Y | 81X1024 | Min:-0.182 Max:0.175 | 0.01 | 0.0001 |
| student.roi_head.bbox_head.fc_cls.bias | Y | 81 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.roi_head.bbox_head.fc_reg.weight | Y | 320X1024 | Min:-0.162 Max:0.198 | 0.01 | 0.0001 |
| student.roi_head.bbox_head.fc_reg.bias | Y | 320 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.roi_head.bbox_head.shared_fcs.0.weight | Y | 1024X12544 | Min:-0.065 Max:0.067 | 0.01 | 0.0001 |
| student.roi_head.bbox_head.shared_fcs.0.bias | Y | 1024 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
| student.roi_head.bbox_head.shared_fcs.1.weight | Y | 1024X1024 | Min:-0.150 Max:0.146 | 0.01 | 0.0001 |
| student.roi_head.bbox_head.shared_fcs.1.bias | Y | 1024 | Min:0.000 Max:0.000 | 0.01 | 0.0001 |
+------------------------------------------------+-----------+---------------+-----------------------+------+--------+
wandb: Currently logged in as: lfgp (use `wandb login --relogin` to force relogin)
wandb: Tracking run with wandb version 0.12.6
wandb: Syncing run soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k
wandb: View project at https://wandb.ai/lfgp/pre_release
wandb: View run at https://wandb.ai/lfgp/pre_release/runs/2rst9oyn
wandb: Run data is saved locally in /mnt/c/Users/Francisco Pereira/Desktop/IST/10º semestre/algoritmos/SoftTeacher/wandb/run-20211110_141125-2rst9oyn
wandb: Run `wandb offline` to turn off syncing.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 0 (pid: 3352) of binary: /home/lfgp/miniconda3/bin/python
Traceback (most recent call last):
File "/home/lfgp/miniconda3/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/lfgp/miniconda3/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/lfgp/miniconda3/lib/python3.9/site-packages/torch/distributed/launch.py", line 193, in <module>
main()
File "/home/lfgp/miniconda3/lib/python3.9/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/home/lfgp/miniconda3/lib/python3.9/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/home/lfgp/miniconda3/lib/python3.9/site-packages/torch/distributed/run.py", line 710, in run
elastic_launch(
File "/home/lfgp/miniconda3/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/lfgp/miniconda3/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
=====================================================
tools/train.py FAILED
-----------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
-----------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2021-11-10_14:13:43
host : LAPTOP-VI0T98FT.localdomain
rank : 0 (local_rank: 0)
exitcode : -9 (pid: 3352)
error_file: <N/A>
traceback : Signal 9 (SIGKILL) received by PID 3352
=====================================================
Solution: KeyError: 'SemiBalanceSampler is not in the sampler registry'
(duplicate of https://github.com/microsoft/SoftTeacher/issues/58)
If you are seeing the error above, you are using an invalid sampler configuration in your config file which has not implemented for your launch mode. You need to override or change the existing config to fix this problem.
Create a new configuration using below content and use it in train command. (you might need to change the _base_)
_base_="soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py"
data = dict(
sampler=dict(
train=dict(
_delete_=True,
type="GroupSampler"
)
),
)
But if you are trying to train on a single GPU, below command should work as mention in https://github.com/microsoft/SoftTeacher/issues/55#issuecomment-950300155 and you don't need to turn off distributed mode.
bash tools/dist_train.sh configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py
@MendelXu consider closing this thread.
Greetings!
one more thing, ensure that in init file for the sampler, you have not commented anything out
Did u solve the problem? i met the same
Did you read https://github.com/microsoft/SoftTeacher/issues/87#issuecomment-1074357271 carefully?
@lahiiru what is the difference between GroupSampler and SemiBalanceSampler?
By this way, i solved the problem. But i met a new problem, as follows:
Traceback (most recent call last): File "tools/train.py", line 200, in <module> main() File "tools/train.py", line 188, in main train_detector( File "/home/yyy/log/4.semi-supervised/SoftTeacher-main/ssod/apis/train.py", line 206, in train_detector runner.run(data_loaders, cfg.workflow) File "/home/yyy/miniconda3/envs/mmdet2x/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 144, in run iter_runner(iter_loaders[i], **kwargs) File "/home/yyy/miniconda3/envs/mmdet2x/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 70, in train self.call_hook('after_train_iter') File "/home/yyy/miniconda3/envs/mmdet2x/lib/python3.8/site-packages/mmcv/runner/base_runner.py", line 317, in call_hook getattr(hook, fn_name)(self) File "/home/yyy/log/4.semi-supervised/SoftTeacher-main/ssod/utils/hooks/submodules_evaluation.py", line 37, in after_train_iter self._do_evaluate(runner) File "/home/yyy/log/4.semi-supervised/SoftTeacher-main/ssod/utils/hooks/submodules_evaluation.py", line 51, in _do_evaluate dist.broadcast(module.running_var, 0) File "/home/yyy/miniconda3/envs/mmdet2x/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1186, in broadcast default_pg = _get_default_group() File "/home/yyy/miniconda3/envs/mmdet2x/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 429, in _get_default_group raise RuntimeError( RuntimeError: Default process group has not been initialized, please make sure to call init_process_group. wandb: Waiting for W&B process to finish... (failed 1).
How can i solve the problem?
Refer to this: # sampler=dict( # train=dict( # type="DistributedGroupSemiBalanceSampler", # if is distributed training，a prefix " DistributedGroup" will be added to "SemiBalanceSampler". # sample_ratio=[1, 4], # by_prob=True, # # at_least_one=True, # epoch_length=7330, # ) # ),
