Input shape ignored when converting from milinternal

[ayes-vjanssen]

❓Question

I am building a custom MIL program using the Python Builder class for MIL. As stated in the coremltools API Reference I am starting my custom program as follows:

import coremltools as ct
from coremltools.converters.mil import Builder as mb
@mb.program(input_specs=[mb.TensorSpec(shape=(1,3,640,640))])
def prog(a):
    return mb.add(x=a, y=2)

How can I assign a flexible input/enumerated input shape to the program?

It seems that converting the program to a MLPackage using ct.convert() and using the attributes inputs and/or outputs do not affect the converted model. In the code snippet below I specifically renamed the input name to new_a. This code snippet will work just fine and will convert the program without any issues. However, if I open the MLProgram in Xcode and look at the Predictions tab I cannot see the changes applied to the input (see image bellow).

Is there a way to give the program a flexible input?

Full code sample:

import coremltools as ct
from coremltools.converters.mil import Builder as mb

@mb.program(input_specs=[mb.TensorSpec(shape=(1,3,640,640))])
def prog(a):
    return mb.add(x=a, y=2.0)


# Set the input_shape to use EnumeratedShapes.
input_shape = ct.EnumeratedShapes(shapes=[[1, 3, 640, 640],
                                          [1, 3, 320, 320]],
                                          default=[1, 3, 640, 640])
mlmodel = ct.convert(prog,
    convert_to="mlprogram",
    inputs=[ct.TensorType(name="a", shape=input_shape)],
    compute_precision=ct.precision.FLOAT16,)
mlmodel.save("simple_test.mlpackage")

Environment:

• coremltools version: 7.0
• macOS Ventura 13.3.1 (MacBook Pro M1 Pro)

[jakesabathia2]

@ayes-vjanssen thanks for using coremltools. This is a good question, in fact, coremltools doesn't support inputs when then source model is milinternal, I believe this will be a useful feature request. But in short, for now, the issue you are facing is expected.

[ayes-vjanssen]

@jakesabathia2 Thanks for checking! Does this also mean that for now it is impossible to convert a MIL Program to a MLProgram that has an Image as expected input? I cannot seem to find a way to specify an image as input spec in the line @mb.program(input_specs=[mb.TensorSpec(shape=(1,3,640,640))]).

[jakesabathia2]

@ayes-vjanssen You are correct, right now the ImageType input can only be supported through pytorch / TF conversion

[ayes-vjanssen]

Before closing this issue and handing in a feature request I have a final question. I was able to modify the CoreML model after converting from MIL Program to ML Program as follows:

mlmodel = ct.convert(prog,
    convert_to="mlprogram",
    compute_precision=ct.precision.FLOAT16,)

   
spec = mlmodel.get_spec()
input = spec.description.input[0]
input.type.imageType.colorSpace = ct.proto.FeatureTypes_pb2.ImageFeatureType.RGB
input.type.imageType.height = 640
input.type.imageType.width = 640

mlmodel = ct.models.model.MLModel(spec, weights_dir=mlmodel.weights_dir)
mlmodel.save("simple_test.mlpackage")

This seems to give the expected result looking at the Xcode Predictions tab. @jakesabathia2 Is this a legit way of making sure the input is of type Image?