mmdetection
mmdetection copied to clipboard
open-mmlab
About MMDetWandbHook
when i use WandbLoggerHook, it's ok! Replace by MMDetWandbHook, the error likes this.
Traceback (most recent call last):
File "E:/mm250/tools/trains/train.py", line 237, in
what command and config did you use? Please use issue template
- I have searched related issues but cannot get the expected help.
- I have read the FAQ documentation but cannot get the expected help.
- The bug has not been fixed in the latest version.
Describe the bug WandbLoggerHook is used in log_config, but it has some errors. rles = maskUtils.frPyObjects(polygons, height, width) File "pycocotools/_mask.pyx", line 293, in pycocotools._mask.frPyObjects IndexError: list index out of range
Reproduction
- What command or script did you run? python train.py --config /public/home/cx/mm251/tools/config/csp1-ms-cos.py --work-dir exp/ms-cos
A placeholder for the command.
- Did you make any modifications on the code or config? Did you understand what you have modified?
- What dataset did you use?
Environment
- Please run
python mmdet/utils/collect_env.pyto collect necessary environment information and paste it here.
sys.platform: linux Python: 3.8.12 (default, Oct 12 2021, 13:49:34) [GCC 7.5.0] CUDA available: True GPU 0: Tesla V100-PCIE-32GB CUDA_HOME: /usr/local/cuda-10.1/ GCC: gcc (GCC) 5.5.0 PyTorch: 1.8.1+cu101 PyTorch compiling details: PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 10.1
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70
- CuDNN 7.6.3
- Magma 2.5.2
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.1, CUDNN_VERSION=7.6.3, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
TorchVision: 0.9.1+cu101 OpenCV: 4.6.0 MMCV: 1.5.0 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 10.1 MMDetection: 2.25.1+3b72b12 3. You may add addition that may be helpful for locating the problem, such as
- How you installed PyTorch [e.g., pip, conda, source]
- Other environment variables that may be related (such as
$PATH,$LD_LIBRARY_PATH,$PYTHONPATH, etc.) myconfig is here! conf.txt
Error traceback If applicable, paste the error trackback here. Errors are shown above.
A placeholder for trackback.
Bug fix If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!
Seems this is a pycocotools error, have you checked your data annotations?
Seems this is a pycocotools error, have you checked your data annotations?
yeah,n I have checked it, and everything is ok when i remove the mmdetWandbHook in log_config, or replace it by 'WandbLoogerHook'. log_config = dict(interval=50, hooks=[ dict(type='TextLoggerHook'), dict( type='MMDetWandbHook', #WandbLoogerHook is ok init_kwargs=dict(entity='cxt5', project='dw2ms'), interval=50, log_checkpoint=True, log_checkpoint_metadata=True ) ])
Hey @chunxt will you be able to share a colab or a GitHub gist to reproduce the issue?
I am also facing a similar issue, which got solved by replacing 'MMDetWandbHook' with 'WandbLoogerHook'. Here is the error log
##################
"
wandb: Run wandb offline to turn off syncing.
wandb: Syncing run desert-leaf-12
wandb: ⭐️ View project at https://wandb.ai/mmwhale/mmwhale
wandb: 🚀 View run at https://wandb.ai/mmwhale/mmwhale/runs/1isubsvx
2022-09-30 15:29:27,819 - mmdet - WARNING - The num_eval_images (74) is greater than the total number of validation samples (74). The complete validation dataset will be logged.
Traceback (most recent call last):
File "tools/train.py", line 253, in
"
@ayulockin @BIGWangYuDong
I see that when your annotation doesn't have any segmentation masks, this error is produced.
I think this has something to do with 'MMDetWandbHook' hook is trying to get segmentation masks, but as the masks are not available in the annotation; pycocotools._mask.frPyObjects throws the error Exception: input type is not supported. (see here)
It's not supposed to go inside the if loop (line 379) but still it goes in somehow (i checked it using print statement)
I request the wandb and mmdetection to solve this issue.