(ValueError: embedde null bype) when run tools/dist_test.sh in more than one GPU

[Taited]

Error Report

07/25 17:33:21 - mmengine - INF0 - Iter(test) [1550/4020] eta: 1:57:23 time: 3.3139 data_time: 2.4773 memory: 5238 [35/1829] [E ProcessGroupNCCL. cpp:587] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(0pType=BR0ADCAST, Timeout(ms)=1800000) ran fo r 1803729 milliseconds before timing out. Traceback (most recent call last): File "./tools/test. py", line 149 , in main() File "./tools/test. py", line 145 , in main runner.test() File "/home/liangyuan/,conda/envs/mmdetection/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1781, in test metrics = self, test_loop.run() # type: ignore File "/home/liangyuan/.conda/envs/mmdetection/lib/python3.8/site-packages/mmengine/runner/loops.py", line 438, in run metrics = self.evaluator. evaluate(len(self.dataloader.dataset)) File "/home/liangyuan/.conda/envs/mmdetection/lib/python3.8/site-packages/mmengine/evaluator/evaluator.py", line 79, in evaluate results = metric. evaluate (size) File "/home/liangyuan/.conda/envs/mmdetection/lib/python3.8/site-packages/mmengine/evaluator/metric.py", line 122, in evaluate results $=$ collect_results ( File "/home/liangyuan/.conda/envs/mmdetection/lib/python3.8/site-packages/mmengine/dist/dist.py", line 931, in collect_results return collect_results_cpu(results, size, tmpdir) File "/home/liangyuan/.conda/envs/mmdetection/lib/python3.8/site-packages/mmengine/dist/dist.py", line 990, in collect_results_cpu with open(osp.join(tmpdir, f'part_{rank}.pkl'), 'wb') as f: # type: ignore ValueError: embedded null byte [E ProcessGroupNCCL. cpp:341] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down. terminate called after throwing an instance of 'std:: runtime_error' what (): [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=BR0ADCAST, Timeout(ms)=1800000) ran for 1803729 millisec onds before timing out. WARNING: torch.distributed.elastic.multiprocessing.api:Sending process 2146061 closing signal SIGTERM ERROR: torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 1 (pid: 2146065) of binary: /home/liangyuan/.conda/e nvs/mmdetection/bin/python Traceback (most recent call last)

Log File

2023/07/25 16:19:06 - mmengine - INFO -

System environment: sys.platform: linux Python: 3.8.17 (default, Jul 5 2023, 21:04:15) [GCC 11.2.0] CUDA available: True numpy_random_seed: 1812757829 GPU 0,1: NVIDIA GeForce RTX 3080 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 11.1, V11.1.74 GCC: gcc (Ubuntu 9.5.0-1ubuntu1~22.04) 9.5.0 PyTorch: 1.10.0+cu111 PyTorch compiling details: PyTorch built with:

• GCC 7.3

• C++ Version: 201402

• Intel(R) oneAPI Math Kernel Library Version 2023.1-Product Build 20230303 for Intel(R) 64 architecture applications

• Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)

• OpenMP 201511 (a.k.a. OpenMP 4.5)

• LAPACK is enabled (usually provided by MKL)

• NNPACK is enabled

• CPU capability usage: AVX512

• CUDA Runtime 11.1

• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86

• CuDNN 8.0.5

• Magma 2.5.2

• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

  TorchVision: 0.11.0+cu111 OpenCV: 4.8.0 MMEngine: 0.8.1

Runtime environment: cudnn_benchmark: False mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0} dist_cfg: {'backend': 'nccl'} seed: 1812757829 Distributed launcher: pytorch Distributed training: True GPU number: 2

2023/07/25 16:19:07 - mmengine - INFO - Config: dataset_type = 'DeepFashion2Dataset' data_root = 'data/deepfashion2/' backend_args = None train_pipeline = [ dict(type='LoadImageFromFile', to_float32=True, backend_args=None), dict(type='LoadAnnotations', with_bbox=True, with_mask=True), dict(type='RandomFlip', prob=0.5), dict( type='RandomResize', scale=( 384, 384, ), ratio_range=( 0.1, 2.0, ), resize_type='Resize', keep_ratio=True), dict( type='RandomCrop', crop_size=( 384, 384, ), crop_type='absolute', recompute_bbox=True, allow_negative_crop=True), dict( type='FilterAnnotations', min_gt_bbox_wh=( 1e-05, 1e-05, ), by_mask=True), dict(type='PackDetInputs'), ] test_pipeline = [ dict(type='LoadImageFromFile', to_float32=True, backend_args=None), dict(type='Resize', scale=( 1333, 880, ), keep_ratio=True), dict(type='LoadAnnotations', with_bbox=True, with_mask=True), dict( type='PackDetInputs', meta_keys=( 'img_id', 'img_path', 'ori_shape', 'img_shape', 'scale_factor', )), ] train_dataloader = dict( batch_size=8, num_workers=4, persistent_workers=True, sampler=dict(type='DefaultSampler', shuffle=True), batch_sampler=dict(type='AspectRatioBatchSampler'), dataset=dict( type='DeepFashion2Dataset', data_root='data/deepfashion2/', ann_file='annotations/instances_train2017.json', data_prefix=dict( img='train2017/', seg='annotations/panoptic_train2017/'), filter_cfg=dict(filter_empty_gt=True, min_size=32), pipeline=[ dict(type='LoadImageFromFile', to_float32=True, backend_args=None), dict(type='LoadAnnotations', with_bbox=True, with_mask=True), dict(type='RandomFlip', prob=0.5), dict( type='RandomResize', scale=( 384, 384, ), ratio_range=( 0.1, 2.0, ), resize_type='Resize', keep_ratio=True), dict( type='RandomCrop', crop_size=( 384, 384, ), crop_type='absolute', recompute_bbox=True, allow_negative_crop=True), dict( type='FilterAnnotations', min_gt_bbox_wh=( 1e-05, 1e-05, ), by_mask=True), dict(type='PackDetInputs'), ], backend_args=None), pin_memory=True) val_dataloader = dict( batch_size=4, num_workers=1, persistent_workers=True, drop_last=False, sampler=dict(type='DefaultSampler', shuffle=False), dataset=dict( type='DeepFashion2Dataset', data_root='data/deepfashion2/', ann_file='annotations/less_instances_val2017.json', data_prefix=dict(img='val2017/', seg='annotations/panoptic_val2017/'), test_mode=True, pipeline=[ dict(type='LoadImageFromFile', to_float32=True, backend_args=None), dict(type='Resize', scale=( 384, 384, ), keep_ratio=True), dict(type='LoadAnnotations', with_bbox=True, with_mask=True), dict( type='PackDetInputs', meta_keys=( 'img_id', 'img_path', 'ori_shape', 'img_shape', 'scale_factor', )), ], backend_args=None), pin_memory=True) test_dataloader = dict( batch_size=4, num_workers=4, persistent_workers=True, drop_last=False, sampler=dict(type='DefaultSampler', shuffle=False), dataset=dict( type='DeepFashion2Dataset', data_root='data/deepfashion2/', ann_file='annotations/instances_val2017.json', data_prefix=dict(img='val2017/', seg='annotations/panoptic_val2017/'), test_mode=True, pipeline=[ dict(type='LoadImageFromFile', to_float32=True, backend_args=None), dict(type='Resize', scale=( 1333, 880, ), keep_ratio=True), dict(type='LoadAnnotations', with_bbox=True, with_mask=True), dict( type='PackDetInputs', meta_keys=( 'img_id', 'img_path', 'ori_shape', 'img_shape', 'scale_factor', )), ], backend_args=None), pin_memory=True) val_evaluator = dict( type='CocoMetric', ann_file='data/deepfashion2/annotations/less_instances_val2017.json', metric=[ 'bbox', 'segm', ], format_only=False, backend_args=None) test_evaluator = dict( type='CocoMetric', ann_file='data/deepfashion2/annotations/instances_val2017.json', metric=[ 'bbox', 'segm', ], format_only=False, backend_args=None) default_scope = 'mmdet' default_hooks = dict( timer=dict(type='IterTimerHook'), logger=dict(type='LoggerHook', interval=50), param_scheduler=dict(type='ParamSchedulerHook'), checkpoint=dict( type='CheckpointHook', interval=5000, by_epoch=False, save_last=True, max_keep_ckpts=3), sampler_seed=dict(type='DistSamplerSeedHook'), visualization=dict( type='DetVisualizationHook', draw=True, show=False, interval=5000)) env_cfg = dict( cudnn_benchmark=False, mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0), dist_cfg=dict(backend='nccl')) vis_backends = [ dict(type='LocalVisBackend'), dict(type='TensorboardVisBackend'), ] visualizer = dict( type='DetLocalVisualizer', vis_backends=[ dict(type='LocalVisBackend'), dict(type='TensorboardVisBackend'), ], name='visualizer') log_processor = dict(type='LogProcessor', window_size=50, by_epoch=False) log_level = 'INFO' load_from = 'work_dirs/5_loss_cls-mask2former_swin-t-p4-w7-384_8xb2-lsj-15e_deepfashion2/iter_183225.pth' resume = False image_size = ( 384, 384, ) batch_augments = [ dict( type='BatchFixedSizePad', size=( 384, 384, ), img_pad_value=0, pad_mask=True, mask_pad_value=0, pad_seg=False), ] data_preprocessor = dict( type='DetDataPreprocessor', mean=[ 123.675, 116.28, 103.53, ], std=[ 58.395, 57.12, 57.375, ], bgr_to_rgb=True, pad_size_divisor=32, pad_mask=True, mask_pad_value=0, pad_seg=False, seg_pad_value=255, batch_augments=[ dict( type='BatchFixedSizePad', size=( 384, 384, ), img_pad_value=0, pad_mask=True, mask_pad_value=0, pad_seg=False), ]) num_things_classes = 13 num_stuff_classes = 0 num_classes = 13 model = dict( type='Mask2Former', data_preprocessor=dict( type='DetDataPreprocessor', mean=[ 123.675, 116.28, 103.53, ], std=[ 58.395, 57.12, 57.375, ], bgr_to_rgb=True, pad_size_divisor=32, pad_mask=True, mask_pad_value=0, pad_seg=False, seg_pad_value=255, batch_augments=[ dict( type='BatchFixedSizePad', size=( 384, 384, ), img_pad_value=0, pad_mask=True, mask_pad_value=0, pad_seg=False), ]), backbone=dict( type='SwinTransformer', embed_dims=96, depths=[ 2, 2, 6, 2, ], num_heads=[ 3, 6, 12, 24, ], window_size=7, mlp_ratio=4, qkv_bias=True, qk_scale=None, drop_rate=0.0, attn_drop_rate=0.0, drop_path_rate=0.3, patch_norm=True, out_indices=( 0, 1, 2, 3, ), with_cp=False, convert_weights=True, frozen_stages=-1, init_cfg=dict( type='Pretrained', checkpoint= 'https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pth' )), panoptic_head=dict( type='Mask2FormerHead', in_channels=[ 96, 192, 384, 768, ], strides=[ 4, 8, 16, 32, ], feat_channels=256, out_channels=256, num_things_classes=13, num_stuff_classes=0, num_queries=100, num_transformer_feat_level=3, pixel_decoder=dict( type='MSDeformAttnPixelDecoder', num_outs=3, norm_cfg=dict(type='GN', num_groups=32), act_cfg=dict(type='ReLU'), encoder=dict( num_layers=6, layer_cfg=dict( self_attn_cfg=dict( embed_dims=256, num_heads=8, num_levels=3, num_points=4, dropout=0.0, batch_first=True), ffn_cfg=dict( embed_dims=256, feedforward_channels=1024, num_fcs=2, ffn_drop=0.0, act_cfg=dict(type='ReLU', inplace=True)))), positional_encoding=dict(num_feats=128, normalize=True)), enforce_decoder_input_project=False, positional_encoding=dict(num_feats=128, normalize=True), transformer_decoder=dict( return_intermediate=True, num_layers=9, layer_cfg=dict( self_attn_cfg=dict( embed_dims=256, num_heads=8, dropout=0.0, batch_first=True), cross_attn_cfg=dict( embed_dims=256, num_heads=8, dropout=0.0, batch_first=True), ffn_cfg=dict( embed_dims=256, feedforward_channels=2048, num_fcs=2, ffn_drop=0.0, act_cfg=dict(type='ReLU', inplace=True))), init_cfg=None), loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=False, loss_weight=5.0, reduction='mean', class_weight=[ 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.1, ]), loss_mask=dict( type='CrossEntropyLoss', use_sigmoid=True, reduction='mean', loss_weight=5.0), loss_dice=dict( type='DiceLoss', use_sigmoid=True, activate=True, reduction='mean', naive_dice=True, eps=1.0, loss_weight=5.0)), panoptic_fusion_head=dict( type='MaskFormerFusionHead', num_things_classes=13, num_stuff_classes=0, loss_panoptic=None, init_cfg=None), train_cfg=dict( num_points=12544, oversample_ratio=3.0, importance_sample_ratio=0.75, assigner=dict( type='HungarianAssigner', match_costs=[ dict(type='ClassificationCost', weight=2.0), dict( type='CrossEntropyLossCost', weight=5.0, use_sigmoid=True), dict(type='DiceCost', weight=5.0, pred_act=True, eps=1.0), ]), sampler=dict(type='MaskPseudoSampler')), test_cfg=dict( panoptic_on=False, semantic_on=False, instance_on=True, max_per_image=50, iou_thr=0.8, filter_low_score=True), init_cfg=None) embed_multi = dict(lr_mult=1.0, decay_mult=0.0) optim_wrapper = dict( type='OptimWrapper', optimizer=dict( type='AdamW', lr=0.0001, weight_decay=0.05, eps=1e-08, betas=( 0.9, 0.999, )), paramwise_cfg=dict( custom_keys=dict({ 'backbone': dict(lr_mult=0.1, decay_mult=1.0), 'query_embed': dict(lr_mult=1.0, decay_mult=0.0), 'query_feat': dict(lr_mult=1.0, decay_mult=0.0), 'level_embed': dict(lr_mult=1.0, decay_mult=0.0), 'backbone.patch_embed.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.norm': dict(lr_mult=0.1, decay_mult=0.0), 'absolute_pos_embed': dict(lr_mult=0.1, decay_mult=0.0), 'relative_position_bias_table': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.0.blocks.0.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.0.blocks.1.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.1.blocks.0.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.1.blocks.1.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.2.blocks.0.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.2.blocks.1.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.2.blocks.2.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.2.blocks.3.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.2.blocks.4.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.2.blocks.5.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.3.blocks.0.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.3.blocks.1.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.0.downsample.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.1.downsample.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.2.downsample.norm': dict(lr_mult=0.1, decay_mult=0.0) }), norm_decay_mult=0.0), clip_grad=dict(max_norm=0.01, norm_type=2)) max_iters = 183225 param_scheduler = dict( type='MultiStepLR', begin=0, end=183225, by_epoch=False, milestones=[ 162867, 176438, ], gamma=0.1) interval = 5000 dynamic_intervals = [ ( 180001, 183225, ), ] train_cfg = dict( type='IterBasedTrainLoop', max_iters=183225, val_interval=5000, dynamic_intervals=[ ( 180001, 183225, ), ]) val_cfg = dict(type='ValLoop') test_cfg = dict(type='TestLoop') auto_scale_lr = dict(enable=True, base_batch_size=16) pretrained = 'https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pth' depths = [ 2, 2, 6, 2, ] backbone_norm_multi = dict(lr_mult=0.1, decay_mult=0.0) backbone_embed_multi = dict(lr_mult=0.1, decay_mult=0.0) custom_keys = dict({ 'backbone': dict(lr_mult=0.1, decay_mult=1.0), 'backbone.patch_embed.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.norm': dict(lr_mult=0.1, decay_mult=0.0), 'absolute_pos_embed': dict(lr_mult=0.1, decay_mult=0.0), 'relative_position_bias_table': dict(lr_mult=0.1, decay_mult=0.0), 'query_embed': dict(lr_mult=1.0, decay_mult=0.0), 'query_feat': dict(lr_mult=1.0, decay_mult=0.0), 'level_embed': dict(lr_mult=1.0, decay_mult=0.0), 'backbone.stages.0.blocks.0.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.0.blocks.1.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.1.blocks.0.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.1.blocks.1.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.2.blocks.0.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.2.blocks.1.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.2.blocks.2.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.2.blocks.3.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.2.blocks.4.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.2.blocks.5.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.3.blocks.0.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.3.blocks.1.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.0.downsample.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.1.downsample.norm': dict(lr_mult=0.1, decay_mult=0.0), 'backbone.stages.2.downsample.norm': dict(lr_mult=0.1, decay_mult=0.0) }) work_dir = 'work_dirs/5_loss_cls-mask2former_swin-t-p4-w7-384_8xb2-lsj-15e_deepfashion2' launcher = 'pytorch'

2023/07/25 16:19:09 - mmengine - INFO - Hooks will be executed in the following order: before_run: (VERY_HIGH ) RuntimeInfoHook
(BELOW_NORMAL) LoggerHook

before_train: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(VERY_LOW ) CheckpointHook

before_train_epoch: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(NORMAL ) DistSamplerSeedHook

before_train_iter: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook

after_train_iter: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook

after_train_epoch: (NORMAL ) IterTimerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook

before_val_epoch: (NORMAL ) IterTimerHook

before_val_iter: (NORMAL ) IterTimerHook

after_val_iter: (NORMAL ) IterTimerHook
(NORMAL ) DetVisualizationHook
(BELOW_NORMAL) LoggerHook

after_val_epoch: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook

after_train: (VERY_LOW ) CheckpointHook

before_test_epoch: (NORMAL ) IterTimerHook

before_test_iter: (NORMAL ) IterTimerHook

after_test_iter: (NORMAL ) IterTimerHook
(NORMAL ) DetVisualizationHook
(BELOW_NORMAL) LoggerHook

after_test_epoch: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook

after_run: (BELOW_NORMAL) LoggerHook

2023/07/25 16:19:38 - mmengine - INFO - Load checkpoint from work_dirs/5_loss_cls-mask2former_swin-t-p4-w7-384_8xb2-lsj-15e_deepfashion2/iter_183225.pth 2023/07/25 16:21:53 - mmengine - INFO - Iter(test) [ 50/4020] eta: 2:57:03 time: 2.6759 data_time: 1.9705 memory: 5238
2023/07/25 16:24:09 - mmengine - INFO - Iter(test) [ 100/4020] eta: 2:55:37 time: 2.7002 data_time: 2.0142 memory: 5238
2023/07/25 16:27:01 - mmengine - INFO - Iter(test) [ 150/4020] eta: 3:09:52 time: 3.4554 data_time: 2.7391 memory: 4658
2023/07/25 16:30:39 - mmengine - INFO - Iter(test) [ 200/4020] eta: 3:27:24 time: 4.1988 data_time: 3.5113 memory: 5238
2023/07/25 16:33:55 - mmengine - INFO - Iter(test) [ 250/4020] eta: 3:34:58 time: 4.0761 data_time: 3.4286 memory: 4658
2023/07/25 16:36:55 - mmengine - INFO - Iter(test) [ 300/4020] eta: 3:34:03 time: 3.6082 data_time: 2.9394 memory: 5238
2023/07/25 16:39:13 - mmengine - INFO - Iter(test) [ 350/4020] eta: 3:25:03 time: 2.7517 data_time: 2.0882 memory: 5238
2023/07/25 16:41:14 - mmengine - INFO - Iter(test) [ 400/4020] eta: 3:15:18 time: 2.4298 data_time: 1.7766 memory: 4774
2023/07/25 16:43:26 - mmengine - INFO - Iter(test) [ 450/4020] eta: 3:08:27 time: 2.6112 data_time: 1.9413 memory: 5238
2023/07/25 16:45:32 - mmengine - INFO - Iter(test) [ 500/4020] eta: 3:02:09 time: 2.5415 data_time: 1.8723 memory: 5238
2023/07/25 16:47:30 - mmengine - INFO - Iter(test) [ 550/4020] eta: 2:55:41 time: 2.3690 data_time: 1.7351 memory: 5238
2023/07/25 16:49:41 - mmengine - INFO - Iter(test) [ 600/4020] eta: 2:51:08 time: 2.6129 data_time: 1.9394 memory: 5238
2023/07/25 16:51:49 - mmengine - INFO - Iter(test) [ 650/4020] eta: 2:46:44 time: 2.5628 data_time: 1.8846 memory: 5238
2023/07/25 16:54:03 - mmengine - INFO - Iter(test) [ 700/4020] eta: 2:43:07 time: 2.6784 data_time: 1.9902 memory: 5238
2023/07/25 16:56:06 - mmengine - INFO - Iter(test) [ 750/4020] eta: 2:38:56 time: 2.4711 data_time: 1.8114 memory: 5238
2023/07/25 16:58:09 - mmengine - INFO - Iter(test) [ 800/4020] eta: 2:34:54 time: 2.4406 data_time: 1.7943 memory: 5238
2023/07/25 17:00:23 - mmengine - INFO - Iter(test) [ 850/4020] eta: 2:31:53 time: 2.6888 data_time: 1.9951 memory: 5238
2023/07/25 17:02:29 - mmengine - INFO - Iter(test) [ 900/4020] eta: 2:28:27 time: 2.5196 data_time: 1.8664 memory: 5238
2023/07/25 17:04:36 - mmengine - INFO - Iter(test) [ 950/4020] eta: 2:25:13 time: 2.5338 data_time: 1.8710 memory: 5238
2023/07/25 17:06:43 - mmengine - INFO - Iter(test) [1000/4020] eta: 2:22:05 time: 2.5331 data_time: 1.8763 memory: 5238
2023/07/25 17:08:46 - mmengine - INFO - Iter(test) [1050/4020] eta: 2:18:53 time: 2.4678 data_time: 1.8154 memory: 4658
2023/07/25 17:10:56 - mmengine - INFO - Iter(test) [1100/4020] eta: 2:16:06 time: 2.6007 data_time: 1.9349 memory: 5238
2023/07/25 17:13:07 - mmengine - INFO - Iter(test) [1150/4020] eta: 2:13:25 time: 2.6288 data_time: 1.9574 memory: 4774
2023/07/25 17:15:11 - mmengine - INFO - Iter(test) [1200/4020] eta: 2:10:29 time: 2.4761 data_time: 1.8292 memory: 4774
2023/07/25 17:17:14 - mmengine - INFO - Iter(test) [1250/4020] eta: 2:07:36 time: 2.4664 data_time: 1.8240 memory: 4658
2023/07/25 17:19:20 - mmengine - INFO - Iter(test) [1300/4020] eta: 2:04:52 time: 2.5164 data_time: 1.8622 memory: 5238
2023/07/25 17:21:30 - mmengine - INFO - Iter(test) [1350/4020] eta: 2:02:18 time: 2.5911 data_time: 1.9153 memory: 5238
2023/07/25 17:24:28 - mmengine - INFO - Iter(test) [1400/4020] eta: 2:01:16 time: 3.5546 data_time: 2.6781 memory: 5238
2023/07/25 17:27:31 - mmengine - INFO - Iter(test) [1450/4020] eta: 2:00:14 time: 3.6475 data_time: 2.7549 memory: 5238
2023/07/25 17:30:35 - mmengine - INFO - Iter(test) [1500/4020] eta: 1:59:07 time: 3.6797 data_time: 2.7934 memory: 5238
2023/07/25 17:33:21 - mmengine - INFO - Iter(test) [1550/4020] eta: 1:57:23 time: 3.3139 data_time: 2.4773 memory: 5238

[Polarisamoon]

Have you solved this problem? I am also encountering the same issue.

[Dis-Able]

Have you solved this problem? I am also encountering the same issue.