create_d2go.py crashes with: quantized engine FBGEMM is not supported

[YaLTeR]

Fedora 33, Python 3.9.2, installed dependencies in a venv according to https://github.com/pytorch/android-demo-app/tree/master/D2Go#quick-start.

Last lines in the output:

[03/05 10:49:12 d2.data.common]: Serializing 3 elements to byte tensors and concatenating them all ...
[03/05 10:49:12 d2.data.common]: Serialized dataset takes 0.00 MiB
[03/05 10:49:12 d2go.export.api]: The model is not quantized during training, running post training quantization ...
terminate called after throwing an instance of 'c10::Error'
  what():  quantized engine FBGEMM is not supported
Exception raised from setQEngine at /pytorch/aten/src/ATen/Context.cpp:184 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f289547e2f2 in /home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x5b (0x7f289547b67b in /home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/lib/libc10.so)
frame #2: at::Context::setQEngine(c10::QEngine) + 0x145 (0x7f28d02e0525 in /home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #3: THPModule_setQEngine(_object*, _object*) + 0x58 (0x7f28e2136c28 in /home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #35: __libc_start_main + 0xf2 (0x7f28f2c771e2 in /lib64/libc.so.6)

[zhanghang1989]

Suggested by @wat3rBro If running on mobile, could you add the following to the config:

QUANTIZATION:
  BACKEND: "qnnpack"

[YaLTeR]

I'm running create_d2go.py on my PC. Nevertheless, after adding that to config this happens:

[03/09 21:32:01 d2go.export.api]: Converting quantized model qnnpack...
Traceback (most recent call last):
  File "/home/yalter/source/java/android-demo-app/D2Go/create_d2go.py", line 96, in <module>
    test_export_torchvision_format()
  File "/home/yalter/source/java/android-demo-app/D2Go/create_d2go.py", line 80, in test_export_torchvision_format
    predictor_path = convert_and_export_predictor(
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/d2go/export/api.py", line 104, in convert_and_export_predictor
    pytorch_model = torch.quantization.convert(pytorch_model, inplace=False)
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/quantization/quantize.py", line 471, in convert
    _convert(
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/quantization/quantize.py", line 507, in _convert
    _convert(mod, mapping, True,  # inplace
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/quantization/quantize.py", line 507, in _convert
    _convert(mod, mapping, True,  # inplace
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/quantization/quantize.py", line 507, in _convert
    _convert(mod, mapping, True,  # inplace
  [Previous line repeated 1 more time]
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/quantization/quantize.py", line 509, in _convert
    reassign[name] = swap_module(mod, mapping, custom_module_class_mapping)
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/quantization/quantize.py", line 534, in swap_module
    new_mod = mapping[type(mod)].from_float(mod)
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/nn/intrinsic/quantized/modules/conv_relu.py", line 97, in from_float
    return super(ConvReLU2d, cls).from_float(mod)
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/nn/quantized/modules/conv.py", line 418, in from_float
    return _ConvNd.from_float(cls, mod)
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/nn/quantized/modules/conv.py", line 220, in from_float
    return cls.get_qconv(mod, activation_post_process, weight_post_process)
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/nn/quantized/modules/conv.py", line 184, in get_qconv
    act_scale, act_zp = activation_post_process.calculate_qparams()
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/quantization/observer.py", line 965, in calculate_qparams
    new_min, new_max = self._non_linear_param_search()
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/quantization/observer.py", line 838, in _non_linear_param_search
    norm = self._compute_quantization_error(next_start_bin, next_end_bin)
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/quantization/observer.py", line 771, in _compute_quantization_error
    norm += self._get_norm(delta_begin, torch.ones(self.bins) * delta_end, density)
  File "/home/yalter/source/py/d2go/venv/lib64/python3.9/site-packages/torch/quantization/observer.py", line 737, in _get_norm
    return density * norm
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

[WayneLin015]

@YaLTeR add pytorch_model.cpu() after pytorch_model = model_zoo.get(cfg_name, trained=True)

[YaLTeR]

That seems to have made it work (combined with the quantization backend config change).

[AhmedHessuin]

@YaLTeR can you tell me where is the config ?

[zhanghang1989]

Could you use qnnpack as the export mode?

https://github.com/facebookresearch/d2go/blob/6140395fe8558c6c14181ed0afbc11daee1d680c/configs/qat_faster_rcnn_fbnetv3a_C4.yaml#L6-L7