about the loss of Mocov3, no decreasing?

[RobinHan24]

分支

main 分支 (mmpretrain 版本)

描述该错误

I trained my own dataset with MocoV3-resnet50, but the loss decreased from 27 to 23, holding on the number of 23, why?

12/20 09:34:59 - mmengine - INFO - Saving checkpoint at 3562 epochs /mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. warnings.warn( 12/20 09:35:11 - mmengine - INFO - Exp name: mocov3_resnet50_8xb512-amp-coslr-800e_in1k_20231219_222815 12/20 09:35:11 - mmengine - INFO - Epoch(train) [3563][3/3] lr: 2.4223e+00 eta: 0:56:54 time: 2.4053 data_time: 1.6347 memory: 18037 loss: 23.5969 12/20 09:35:11 - mmengine - INFO - Saving checkpoint at 3563 epochs /mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. warnings.warn( 12/20 09:35:22 - mmengine - INFO - Exp name: mocov3_resnet50_8xb512-amp-coslr-800e_in1k_20231219_222815 12/20 09:35:22 - mmengine - INFO - Epoch(train) [3564][3/3] lr: 2.4127e+00 eta: 0:56:47 time: 2.3942 data_time: 1.6182 memory: 18037 loss: 23.5970 12/20 09:35:22 - mmengine - INFO - Saving checkpoint at 3564 epochs /mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. warnings.warn( 12/20 09:35:33 - mmengine - INFO - Exp name: mocov3_resnet50_8xb512-amp-coslr-800e_in1k_20231219_222815 12/20 09:35:33 - mmengine - INFO - Epoch(train) [3565][3/3] lr: 2.4032e+00 eta: 0:56:39 time: 2.3664 data_time: 1.5931 memory: 18037 loss: 23.5975 12/20 09:35:33 - mmengine - INFO - Saving checkpoint at 3565 epochs /mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. warnings.warn( 12/20 09:35:44 - mmengine - INFO - Exp name: mocov3_resnet50_8xb512-amp-coslr-800e_in1k_20231219_222815 12/20 09:35:44 - mmengine - INFO - Epoch(train) [3566][3/3] lr: 2.3936e+00 eta: 0:56:31 time: 2.3551 data_time: 1.5844 memory: 18037 loss: 23.5980 12/20 09:35:44 - mmengine - INFO - Saving checkpoint at 3566 epochs /mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. warnings.warn( 12/20 09:35:56 - mmengine - INFO - Exp name: mocov3_resnet50_8xb512-amp-coslr-800e_in1k_20231219_222815 12/20 09:35:56 - mmengine - INFO - Epoch(train) [3567][3/3] lr: 2.3841e+00 eta: 0:56:23 time: 2.3336 data_time: 1.5614 memory: 18037 loss: 23.5950 12/20 09:35:56 - mmengine - INFO - Saving checkpoint at 3567 epochs /mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. warnings.warn( 12/20 09:36:07 - mmengine - INFO - Exp name: mocov3_resnet50_8xb512-amp-coslr-800e_in1k_20231219_222815 12/20 09:36:07 - mmengine - INFO - Epoch(train) [3568][3/3] lr: 2.3745e+00 eta: 0:56:15 time: 2.4241 data_time: 1.6476 memory: 18037 loss: 23.5938 12/20 09:36:07 - mmengine - INFO - Saving checkpoint at 3568 epochs /mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. warnings.warn( 12/20 09:36:18 - mmengine - INFO - Exp name: mocov3_resnet50_8xb512-amp-coslr-800e_in1k_20231219_222815 12/20 09:36:18 - mmengine - INFO - Epoch(train) [3569][3/3] lr: 2.3650e+00 eta: 0:56:08 time: 2.4099 data_time: 1.6249 memory: 18037 loss: 23.5949 12/20 09:36:18 - mmengine - INFO - Saving checkpoint at 3569 epochs /mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. warnings.warn( 12/20 09:36:29 - mmengine - INFO - Exp name: mocov3_resnet50_8xb512-amp-coslr-800e_in1k_20231219_222815 12/20 09:36:29 - mmengine - INFO - Epoch(train) [3570][3/3] lr: 2.3555e+00 eta: 0:56:00 time: 2.3381 data_time: 1.5578 memory: 18037 loss: 23.5944 12/20 09:36:29 - mmengine - INFO - Saving checkpoint at 3570 epochs /mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. warnings.warn( 12/20 09:36:39 - mmengine - INFO - Exp name: mocov3_resnet50_8xb512-amp-coslr-800e_in1k_20231219_222815 12/20 09:36:39 - mmengine - INFO - Epoch(train) [3571][3/3] lr: 2.3459e+00 eta: 0:55:52 time: 2.1778 data_time: 1.4037 memory: 18037 loss: 23.5939 12/20 09:36:39 - mmengine - INFO - Saving checkpoint at 3571 epochs /mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. warnings.warn( 12/20 09:36:50 - mmengine - INFO - Exp name: mocov3_resnet50_8xb512-amp-coslr-800e_in1k_20231219_222815 12/20 09:36:50 - mmengine - INFO - Epoch(train) [3572][3/3] lr: 2.3364e+00 eta: 0:55:44 time: 2.2672 data_time: 1.4953 memory: 18037 loss: 23.5931 12/20 09:36:50 - mmengine - INFO - Saving checkpoint at 3572 epochs /mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. warnings.warn( 12/20 09:37:02 - mmengine - INFO - Exp name: mocov3_resnet50_8xb512-amp-coslr-800e_in1k_20231219_222815 12/20 09:37:02 - mmengine - INFO - Epoch(train) [3573][3/3] lr: 2.3268e+00 eta: 0:55:36 time: 2.3623 data_time: 1.5885 memory: 18037 loss: 23.5929 12/20 09:37:02 - mmengine - INFO - Saving checkpoint at 3573 epochs

环境信息

{'sys.platform': 'linux', 'Python': '3.9.0 (default, Nov 15 2020, 14:28:56) [GCC 7.3.0]', 'CUDA available': True, 'numpy_random_seed': 2147483648, 'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A10', 'CUDA_HOME': '/usr/local/cuda-11.7', 'NVCC': 'Cuda compilation tools, release 11.7, V11.7.99', 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0', 'PyTorch': '1.13.1+cu117', 'TorchVision': '0.14.1+cu117', 'OpenCV': '4.8.0', 'MMEngine': '0.7.3', 'MMCV': '2.0.0', 'MMPreTrain': '1.0.0rc7+e80418a'}

其他信息

my configure file: mocov3_resnet50_8xb512-amp-coslr-800e_in1k.py base = [ 'imagenet_bs512_mocov3.py', 'default_runtime.py', ]

model settings

temperature = 1.0 model = dict( type='MoCoV3', base_momentum=0.004, # 0.01 for 100e and 300e, 0.004 for 800 and 1000e backbone=dict( type='ResNet', depth=50, norm_cfg=dict(type='SyncBN'), zero_init_residual=False), neck=dict( type='NonLinearNeck', in_channels=2048, hid_channels=4096, out_channels=256, num_layers=2, with_bias=False, with_last_bn=True, with_last_bn_affine=False, with_last_bias=False, with_avg_pool=True), head=dict( type='MoCoV3Head', predictor=dict( type='NonLinearNeck', in_channels=256, hid_channels=4096, out_channels=256, num_layers=2, with_bias=False, with_last_bn=False, with_last_bn_affine=False, with_last_bias=False, with_avg_pool=False), loss=dict(type='CrossEntropyLoss', loss_weight=2 * temperature), temperature=temperature))

optimizer

optim_wrapper = dict( type='AmpOptimWrapper', loss_scale='dynamic', optimizer=dict(type='LARS', lr=4.8, weight_decay=1.5e-6, momentum=0.9), paramwise_cfg=dict( custom_keys={ 'bn': dict(decay_mult=0, lars_exclude=True), 'bias': dict(decay_mult=0, lars_exclude=True), # bn layer in ResNet block downsample module 'downsample.1': dict(decay_mult=0, lars_exclude=True), }), )

learning rate scheduler

param_scheduler = [ dict( type='LinearLR', start_factor=1e-4, by_epoch=True, begin=0, end=10, convert_to_iter_based=True), dict( type='CosineAnnealingLR', T_max=790, by_epoch=True, begin=10, end=4000, convert_to_iter_based=True) ]

runtime settings

train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=4000)

only keeps the latest 3 checkpoints

default_hooks = dict(checkpoint=dict(max_keep_ckpts=3))

NOTE: auto_scale_lr is for automatically scaling LR

based on the actual training batch size.

auto_scale_lr = dict(base_batch_size=4096)

imagenet_bs512_mocov3.py

dataset settings

dataset_type = 'CustomDataset' data_root = 'data/yf5class_old/' data_preprocessor = dict( type='SelfSupDataPreprocessor', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)

view_pipeline1 = [ dict( type='RandomResizedCrop', scale=224, crop_ratio_range=(0.2, 1.), backend='pillow'), dict( type='RandomApply', transforms=[ dict( type='ColorJitter', brightness=0.4, contrast=0.4, saturation=0.2, hue=0.1) ], prob=0.8), dict( type='RandomGrayscale', prob=0.2, keep_channels=True, channel_weights=(0.114, 0.587, 0.2989)), dict( type='GaussianBlur', magnitude_range=(0.1, 2.0), magnitude_std='inf', prob=1.), dict(type='Solarize', thr=128, prob=0.), dict(type='RandomFlip', prob=0.5), ] view_pipeline2 = [ dict( type='RandomResizedCrop', scale=224, crop_ratio_range=(0.2, 1.), backend='pillow'), dict( type='RandomApply', transforms=[ dict( type='ColorJitter', brightness=0.4, contrast=0.4, saturation=0.2, hue=0.1) ], prob=0.8), dict( type='RandomGrayscale', prob=0.2, keep_channels=True, channel_weights=(0.114, 0.587, 0.2989)), dict( type='GaussianBlur', magnitude_range=(0.1, 2.0), magnitude_std='inf', prob=0.1), dict(type='Solarize', thr=128, prob=0.2), dict(type='RandomFlip', prob=0.5), ] train_pipeline = [ dict(type='LoadImageFromFile'), dict( type='MultiView', num_views=[1, 1], transforms=[view_pipeline1, view_pipeline2]), dict(type='PackInputs') ]

train_dataloader = dict( batch_size=192, num_workers=8, persistent_workers=True, pin_memory=True, sampler=dict(type='DefaultSampler', shuffle=True), collate_fn=dict(type='default_collate'), dataset=dict( type='CustomDataset', data_root=data_root, ann_file='', # 我们假定使用子文件夹格式，因此需要将标注文件置空 data_prefix='train', pipeline=train_pipeline))