Failure to Trace MiDaS Model

[rdcdt1]

i try to use ai-hub on midas ( a very populare model to do image to depth map ) but from what i understand it doesn't support to be trace here the code i use

import torch
import urllib.request

# Load MiDaS model
model_type = "DPT_Large"     # MiDaS v3 - Large     (highest accuracy, slowest inference speed)
#model_type = "DPT_Hybrid"   # MiDaS v3 - Hybrid    (medium accuracy, medium inference speed)
#model_type = "MiDaS_small"  # MiDaS v2.1 - Small   (lowest accuracy, highest inference speed)
midas = torch.hub.load("intel-isl/MiDaS", model_type)

# Trace MiDaS model
input_shape = (1, 3, 384, 384)  # Adjust input shape as needed
example_input = torch.rand(input_shape)
traced_midas = torch.jit.trace(midas, example_input)

# Optimize model for the chosen device
device = hub.Device("Samsung Galaxy S23 Ultra")
compile_job = hub.submit_compile_job(
    model=traced_midas,
    name="MyMiDaSModel",
    device=device,
    input_specs=dict(image=input_shape),
)

# Run the model on a hosted device
profile_job = hub.submit_profile_job(
    model=compile_job.get_target_model(),
    device=device,
)

here the log of the execution :

C:\Users\iphone/.cache\torch\hub\intel-isl_MiDaS_master\midas\backbones\vit.py:22: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  gs_old = int(math.sqrt(len(posemb_grid)))
Traceback (most recent call last):
  File "C:\Users\iphone\midas_qualcomm\test.py", line 14, in <module>
    traced_midas = torch.jit.trace(midas, example_input)
  File "C:\Users\iphone\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\jit\_trace.py", line 794, in trace
    return trace_module(
  File "C:\Users\iphone\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\jit\_trace.py", line 1056, in trace_module
    module._c._create_method_from_trace(
  File "C:\Users\iphone\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\iphone\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1488, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "C:\Users\iphone/.cache\torch\hub\intel-isl_MiDaS_master\midas\dpt_depth.py", line 166, in forward
    return super().forward(x).squeeze(dim=1)
  File "C:\Users\iphone/.cache\torch\hub\intel-isl_MiDaS_master\midas\dpt_depth.py", line 114, in forward
    layers = self.forward_transformer(self.pretrained, x)
  File "C:\Users\iphone/.cache\torch\hub\intel-isl_MiDaS_master\midas\backbones\vit.py", line 13, in forward_vit
    return forward_adapted_unflatten(pretrained, x, "forward_flex")
  File "C:\Users\iphone/.cache\torch\hub\intel-isl_MiDaS_master\midas\backbones\utils.py", line 99, in forward_adapted_unflatten
    nn.Unflatten(
  File "C:\Users\iphone\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\flatten.py", line 110, in __init__
    self._require_tuple_int(unflattened_size)
  File "C:\Users\iphone\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\flatten.py", line 133, in _require_tuple_int
    raise TypeError("unflattened_size must be tuple of ints, " +
TypeError: unflattened_size must be tuple of ints, but found element of type Tensor at pos 0

did i do something wrong ?

[bhushan23]

Hi @rdcdt1 seems like Unflatten op is presented with a torch.Tensor instead of integers This is unusual though as C:\Users\iphone/.cache\torch\hub\intel-isl_MiDaS_master\midas\backbones\utils.py seem to have following initialization which should be integer.

Could you please add a breakpoint before line 99 in utils.py and check unflatten input size being provided? this could be an issue with torch version as well (not able to find supported torch version for midas)

[mestrona-3]

Hi @rdcdt1 , in our last AI Hub Models release (0.7.0), Midas is available in repository. https://github.com/quic/ai-hub-models/tree/main/qai_hub_models/models/midas. We recommend you try this version via AI Hub and let us know if you hit any issues!

[rdcdt1]

hi @mestrona-3 sorry for the delay i test the command given but got error. ~/.pyenv/versions/3.10.15/bin/python3.10 -m qai_hub_models.models.midas_quantized.export /home/iphone/.pyenv/versions/3.10.15/bin/python3.10: Error while finding module specification for 'qai_hub_models.models.midas_quantized.export' (ModuleNotFoundError: No module named 'qai_hub_models.models.midas_quantized') i have of course install with the command ~/.pyenv/versions/3.10.15/bin/python3.10 -m pip install "qai_hub_models[midas_quantized]"