Adding enumerated sizes works for single input model but not multiple inputs models.

[MatthieuToulemont]

🐞Describe the bug

Adding enumerated sizes works for single input model but not multiple inputs models. In this issue I use ImageType input but the same happens when using TensorType and Enumerated Shapes.

Trace

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-25-6f1334d000e1> in <module>
      2     {
      3         "x1":Image.fromarray(np.ones((320,320)).astype(np.uint8)),
----> 4         "x2":Image.fromarray(np.ones((320,320)).astype(np.uint8)),
      5     })

~/anaconda3/envs/u2net/lib/python3.7/site-packages/coremltools/models/model.py in predict(self, data, useCPUOnly, **kwargs)
    327 
    328         if self.__proxy__:
--> 329             return self.__proxy__.predict(data, useCPUOnly)
    330         else:
    331             if _macos_version() < (10, 13):

RuntimeError: {
    NSLocalizedDescription = "Error binding image input buffer x1.";
}

To Reproduce

class DummyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = torch.nn.Conv2d(1, 1, kernel_size=3, stride=1, padding=1)
    def forward(self, x):
        x = self.conv(x)
        return x

class DummyModelTwoInputs(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = torch.nn.Conv2d(2, 1, kernel_size=3, stride=1, padding=1)
    def forward(self, x, x2):
        x = self.conv(torch.cat((x, x2), 1))
        return x
dummy_model = DummyModel()
dummy_model_two_inputs = DummyModelTwoInputs()

# Convert Single input model : 
dummy_input = torch.ones((1,1,160,160)).float()
traced_model = torch.jit.trace(dummy_model, [dummy_input])

inputs = [ct.ImageType(name="x1", shape=dummy_input.shape)]
dummy_cml = ct.convert(traced_model, inputs=inputs)
dummy_spec = dummy_cml.get_spec()

# Convert Double input model : 
dummy_input = torch.ones((1,1,160,160)).float()
dummy_input2 = torch.ones((1,1,160,160)).float()
traced_model = torch.jit.trace(dummy_model_two_inputs, [dummy_input, dummy_input2])
inputs = [ct.ImageType(name="x1", shape=dummy_input.shape), ct.ImageType(name="x2", shape=dummy_input2.shape)]
dummy_cml_two_inputs = ct.convert(traced_model, inputs=inputs)
dummy_spec_two_inputs = dummy_cml_two_inputs.get_spec()
dummy_spec_two_inputs.description.output

# Add Enumerated sizes : 
image_sizes = [
      flexible_shape_utils.NeuralNetworkImageSize(320, 320),
      flexible_shape_utils.NeuralNetworkImageSize(640, 640),
      flexible_shape_utils.NeuralNetworkImageSize(640, 920),
      flexible_shape_utils.NeuralNetworkImageSize(512, 512),
]
for key in ["x1"]:
    flexible_shape_utils.add_enumerated_image_sizes(dummy_spec, feature_name=key,
                                                sizes=image_sizes)
for key in ["x1", "x2"]:
    flexible_shape_utils.add_enumerated_image_sizes(dummy_spec_two_inputs, feature_name=key,
                                                sizes=image_sizes)
dummy_with_enum_one_input = ct.models.MLModel(dummy_spec)
dummy_with_enum_two_inputs = ct.models.MLModel(dummy_spec_two_inputs)

Running :

dummy_with_enum_one_input.predict(
    {
        "x1":Image.fromarray(np.ones((320,320)).astype(np.uint8)),
    })['20'].shape

yields : (1, 1, 320, 320)

Whereas running :

dummy_with_enum_two_inputs.predict(
    {
        "x1":Image.fromarray(np.ones((320,320)).astype(np.uint8)),
        "x2":Image.fromarray(np.ones((320,320)).astype(np.uint8)),
    })

yields the error posted above ⬆️

System environment:

• coremltools version : 4.0 (same behavior with 4.1)
• OS : MacOs
• How you install python : pip
• python version : 3.7
• torch : 1.5.0

[TobyRoseman]

@MatthieuTPHR - Thank you for the detailed description. With the following imports, I can reproduce your results.

import torch
import coremltools as ct
from coremltools.models.neural_network import flexible_shape_utils
import numpy as np
from PIL import Image

[MatthieuToulemont]

Great, don't hesitate to message me if you need more information :D

[MatthieuToulemont]

Are there any updates regarding this bug ?

[aseemw]

I just verified that enumerated shapes are only allowed for a single input. For multiple inputs, please use "range" flexibility instead. The error message should be improved.

Also, the recommended approach to mark shapes as flexible is to use ct.EnumeratedShapes and ct.RangeDim during conversion which allows the converter to check whether the model supports flexible input shapes or not. (See https://coremltools.readme.io/docs/flexible-inputs)

[mgstar1021]

Are there any updates regarding this bug ?

Hello, I have been also trying to support dynamic size for u2net model but apple doesn't reply me. Did you solve the problem by using rangedim as Assemw said?

[TobyRoseman]

Using coremltools 6.0, both predict class now run without error.