TypeError: LoadAnnotations.__init__() got an unexpected keyword argument 'with_mask'

[volkov-maxim]

Description of the Issue

I need to run mmpose model after mmdet model for video processing. Runtime errorTypeError: LoadAnnotations.__init__() got an unexpected keyword argument 'with_mask' raised when I called inference_detector() function for mmdetection model.

Simplified script for reproduction

import cv2
from mmdet.apis import init_detector, inference_detector
from mmpose.apis import inference_topdown, init_model
from mmpose.utils import register_all_modules

# Neural networks settings.
# Detection.
det_config = "cascade-mask-rcnn_s101_fpn_syncbn-backbone+head_ms-1x_coco.py"
det_checkpoint = "cascade_mask_rcnn_s101_fpn_syncbn-backbone+head_mstrain_1x_coco_20201005_113243-42607475.pth"
# Pose estimation (wholebody).
pose_config = "td-hm_hrnet-w48_dark-8xb32-210e_coco-wholebody-384x288.py"
pose_checkpoint = "hrnet_w48_coco_wholebody_384x288_dark-f5726563_20200918.pth"

device = "cpu"
print("Used device:", device)

# Initiate the detection model from a config file and a checkpoint file.
det_model = init_detector(det_config, det_checkpoint, device=device)

# Initiate the pose model from a config file and a checkpoint file.
register_all_modules()
pose_model = init_model(pose_config, pose_checkpoint, device=device)

image = cv2.imread("demo.jpg", cv2.IMREAD_COLOR)

mmdet_results = inference_detector(det_model, image)
pose_results = inference_topdown(pose_model, image)

Environment

python -c "from mmcv.utils import collect_env; print(collect_env())"

objc[69973]: Class AppleTypeCRetimerRestoreInfoHelper is implemented in both /usr/lib/libauthinstall.dylib (0x1f009b650) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x10794c598). One of the two will be used. Which one is undefined.
objc[69973]: Class AppleTypeCRetimerFirmwareAggregateRequestCreator is implemented in both /usr/lib/libauthinstall.dylib (0x1f009b6a0) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x10794c5e8). One of the two will be used. Which one is undefined.
objc[69973]: Class AppleTypeCRetimerFirmwareRequestCreator is implemented in both /usr/lib/libauthinstall.dylib (0x1f009b6f0) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x10794c638). One of the two will be used. Which one is undefined.
objc[69973]: Class ATCRTRestoreInfoFTABFile is implemented in both /usr/lib/libauthinstall.dylib (0x1f009b740) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x10794c688). One of the two will be used. Which one is undefined.
objc[69973]: Class AppleTypeCRetimerFirmwareCopier is implemented in both /usr/lib/libauthinstall.dylib (0x1f009b790) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x10794c6d8). One of the two will be used. Which one is undefined.
objc[69973]: Class ATCRTRestoreInfoFTABSubfile is implemented in both /usr/lib/libauthinstall.dylib (0x1f009b7e0) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x10794c728). One of the two will be used. Which one is undefined.
objc[69974]: Class AppleTypeCRetimerRestoreInfoHelper is implemented in both /usr/lib/libauthinstall.dylib (0x1f009b650) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x1047cc598). One of the two will be used. Which one is undefined.
objc[69974]: Class AppleTypeCRetimerFirmwareAggregateRequestCreator is implemented in both /usr/lib/libauthinstall.dylib (0x1f009b6a0) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x1047cc5e8). One of the two will be used. Which one is undefined.
objc[69974]: Class AppleTypeCRetimerFirmwareRequestCreator is implemented in both /usr/lib/libauthinstall.dylib (0x1f009b6f0) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x1047cc638). One of the two will be used. Which one is undefined.
objc[69974]: Class ATCRTRestoreInfoFTABFile is implemented in both /usr/lib/libauthinstall.dylib (0x1f009b740) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x1047cc688). One of the two will be used. Which one is undefined.
objc[69974]: Class AppleTypeCRetimerFirmwareCopier is implemented in both /usr/lib/libauthinstall.dylib (0x1f009b790) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x1047cc6d8). One of the two will be used. Which one is undefined.
objc[69974]: Class ATCRTRestoreInfoFTABSubfile is implemented in both /usr/lib/libauthinstall.dylib (0x1f009b7e0) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x1047cc728). One of the two will be used. Which one is undefined.

OrderedDict([('sys.platform', 'darwin'), ('Python', '3.10.11 (main, Apr  7 2023, 07:24:47) [Clang 14.0.0 (clang-1400.0.29.202)]'), ('CUDA available', False), ('numpy_random_seed', 2147483648), ('GCC', 'Apple clang version 13.0.0 (clang-1300.0.29.3)'), ('PyTorch', '2.0.0'), ('PyTorch compiling details', 'PyTorch built with:\n  - GCC 4.2\n  - C++ Version: 201703\n  - clang 13.1.6\n  - LAPACK is enabled (usually provided by MKL)\n  - NNPACK is enabled\n  - CPU capability usage: NO AVX\n  - Build settings: BLAS_INFO=accelerate, BUILD_TYPE=Release, CXX_COMPILER=/Applications/Xcode_13.3.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -Wno-deprecated-declarations -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_PYTORCH_METAL_EXPORT -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DUSE_COREML_DELEGATE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=braced-scalar-init -Werror=range-loop-construct -Werror=bool-operation -Winconsistent-missing-override -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wvla-extension -Wno-range-loop-analysis -Wno-pass-failed -Wsuggest-override -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wconstant-conversion -Wno-invalid-partial-specialization -Wno-typedef-redefinition -Wno-unused-private-field -Wno-inconsistent-missing-override -Wno-constexpr-not-const -Wno-missing-braces -Wunused-lambda-capture -Wunused-local-typedef -Qunused-arguments -fcolor-diagnostics -fdiagnostics-color=always -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -DUSE_MPS -fno-objc-arc -Wno-unguarded-availability-new -Wno-unused-private-field -Wno-missing-braces -Wno-constexpr-not-const, LAPACK_INFO=accelerate, TORCH_DISABLE_GPU_ASSERTS=OFF, TORCH_VERSION=2.0.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=OFF, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=OFF, USE_ROCM=OFF, \n'), ('TorchVision', '0.15.1'), ('OpenCV', '4.7.0'), ('MMEngine', '0.7.2'), ('MMCV', '2.0.0'), ('MMCV Compiler', 'clang 13.0.0'), ('MMCV CUDA Compiler', 'not available')])

Addition:

• PyTorch installed with pip
• openmim installed with pip
• mmengine, mmcv, mmdet, mmpose installed with mim
• Config and checkpoint files were downloaded with the help of mim
• The demo.jpg can be downloaded from Github.

Error traceback

Used device: cpu
Loads checkpoint by local backend from path: cascade_mask_rcnn_s101_fpn_syncbn-backbone+head_mstrain_1x_coco_20201005_113243-42607475.pth
Loads checkpoint by local backend from path: hrnet_w48_coco_wholebody_384x288_dark-f5726563_20200918.pth
/Users/vmj/Projects/video/lib/python3.10/site-packages/mmpose/datasets/datasets/utils.py:102: UserWarning: The metainfo config file "configs/_base_/datasets/coco_wholebody.py" does not exist. A matched config file "/Users/vmj/Projects/video/lib/python3.10/site-packages/mmpose/.mim/configs/_base_/datasets/coco_wholebody.py" will be used instead.
  warnings.warn(
Traceback (most recent call last):
  File "/Users/vmj/Projects/video/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
TypeError: LoadAnnotations.__init__() got an unexpected keyword argument 'with_mask'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/vmj/Projects/video/extract_2d_coordinates_from_video/example.py", line 26, in <module>
    mmdet_results = inference_detector(det_model, image)
  File "/Users/vmj/Projects/video/lib/python3.10/site-packages/mmdet/apis/inference.py", line 152, in inference_detector
    test_pipeline = Compose(test_pipeline)
  File "/Users/vmj/Projects/video/lib/python3.10/site-packages/mmcv/transforms/wrappers.py", line 66, in __init__
    transform = TRANSFORMS.build(transform)
  File "/Users/vmj/Projects/video/lib/python3.10/site-packages/mmengine/registry/registry.py", line 545, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/Users/vmj/Projects/video/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 135, in build_from_cfg
    raise type(e)(
TypeError: class `LoadAnnotations` in mmcv/transforms/loading.py: LoadAnnotations.__init__() got an unexpected keyword argument 'with_mask'

Imperfect bug fix If to change the place of initialization of models, then an error is not raised on the first frame, but it does on the second frame. In this case on the first frame raised warning The current default scope "mmdet" is not "mmpose", `init_default_scope` will force set the currentdefault scope to "mmpose".

[volkov-maxim]

Found workaround.

Simplified workaround

from mmdet.apis import init_detector, inference_detector
from mmpose.apis import MMPoseInferencer

# Neural networks settings.
# Detection.
det_config = "cascade-mask-rcnn_s101_fpn_syncbn-backbone+head_ms-1x_coco.py"
det_checkpoint = "cascade_mask_rcnn_s101_fpn_syncbn-backbone+head_mstrain_1x_coco_20201005_113243-42607475.pth"
# Pose estimation (wholebody).
pose_config = "td-hm_hrnet-w48_dark-8xb32-210e_coco-wholebody-384x288.py"
pose_checkpoint = "hrnet_w48_coco_wholebody_384x288_dark-f5726563_20200918.pth"

device = "cpu"
print("Used device:", device)

# Initiate the detection model from a config file and a checkpoint file.
det_model = init_detector(det_config, det_checkpoint, device=device)

# Build the inferencer with model config path and checkpoint path/URL.
inferencer = MMPoseInferencer(
    pose2d=pose_config,
    pose2d_weights=pose_checkpoint,
    det_model=det_config,
    det_weights=det_checkpoint,
    det_cat_ids=[0],
    device=device,
)

img_path = "demo.jpg"

mmdet_results = inference_detector(det_model, image)

# The MMPoseInferencer API employs a lazy inference approach,
# creating a prediction generator when given input.
result_generator = inferencer(img_path)

[atechhack]

I have the same issue, calling init_detector before every new frame also seems to work, but is not very efficient.

[markloyman]

The correct way to combine usage of detector model and pose model is with mmpose.utils.adapt_mmdet_pipeline: det_model.cfg = adapt_mmdet_pipeline(det_model.cfg) which converts pipeline types in MMDetection's test dataloader to use the 'mmdet' namespace.

Btw, you can play around with scope using init_default_scope: Add init_default_scope('mmdet') before running inference_detector(..) and init_default_scope('mmpose') before running inference_topdown(..).