coremltools
coremltools copied to clipboard
apple
ImageType causes an error in TensorFlow 2 model conversion with a scale parameter provided as list
🐞Describing the bug
I want to add an input image normalization layer into the resulting CoreML model, which is converted from a TensorFlow 2 model. I'm doing it with ImageType and providing the scale parameters for each input channel as a list. When I call the convert function at the step:
Running MIL backend_mlprogram pipeline
I see the following error:
Incompatible dim 3 in shapes (1, 3, 224, 224) vs. (1, 1, 1, 3)
Stack Trace
2025-03-02 22:35:20.076451: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2025-03-02 22:35:20.076647: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2025-03-02 22:35:20.084452: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 7523 MB memory: -> device: 0, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1
2025-03-02 22:35:21.634132: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2025-03-02 22:35:21.634260: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2025-03-02 22:35:21.648457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 7523 MB memory: -> device: 0, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1
When both 'convert_to' and 'minimum_deployment_target' not specified, 'convert_to' is set to "mlprogram" and 'minimum_deployment_target' is set to ct.target.iOS15 (which is same as ct.target.macOS12). Note: the model will not run on systems older than iOS15/macOS12/watchOS8/tvOS15. In order to make your model run on older system, please set the 'minimum_deployment_target' to iOS14/iOS13. Details please see the link: https://apple.github.io/coremltools/docs-guides/source/target-conversion-formats.html
2025-03-02 22:35:22.786067: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2025-03-02 22:35:22.786239: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2025-03-02 22:35:22.794399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 7523 MB memory: -> device: 0, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1
2025-03-02 22:35:24.339004: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2025-03-02 22:35:24.339128: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2025-03-02 22:35:24.344350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 7523 MB memory: -> device: 0, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1
Running TensorFlow Graph Passes: 0%| | 0/6 [00:00<?, ? passes/s]2025-03-02 22:35:25.448001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 7523 MB memory: -> device: 0, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1
2025-03-02 22:35:25.490373: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:353] MLIR V1 optimization pass is not enabled
Running TensorFlow Graph Passes: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 9.71 passes/s]
Converting TF Frontend ==> MIL Ops: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 426/426 [00:00<00:00, 702.93 ops/s]
Running MIL frontend_tensorflow2 pipeline: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 20.69 passes/s]
Running MIL default pipeline: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 89/89 [00:05<00:00, 16.17 passes/s]
Running MIL backend_mlprogram pipeline: 0%| | 0/12 [00:00<?, ? passes/s]
ERROR - 'mil_backend::insert_image_preprocessing_ops' graph pass produces the following error:
Running MIL backend_mlprogram pipeline: 17%|████████████████████████████████████████▎ | 2/12 [00:00<00:00, 44.50 passes/s]
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[5], line 1
----> 1 coreml_model = ct.convert(keras_model, inputs=[input_type])
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/_converters_entry.py:635, in convert(model, source, inputs, outputs, classifier_config, minimum_deployment_target, convert_to, compute_precision, skip_model_load, compute_units, package_dir, debug, pass_pipeline, states)
632 if len(states) > 0 and exact_source != "pytorch":
633 raise ValueError("'states' can only be passed with pytorch source model.")
--> 635 mlmodel = mil_convert(
636 model,
637 convert_from=exact_source,
638 convert_to=exact_target,
639 inputs=inputs,
640 outputs=outputs_as_tensor_or_image_types, # None or list[ct.ImageType/ct.TensorType]
641 classifier_config=classifier_config,
642 skip_model_load=skip_model_load,
643 compute_units=compute_units,
644 package_dir=package_dir,
645 debug=debug,
646 specification_version=specification_version,
647 main_pipeline=pass_pipeline,
648 use_default_fp16_io=use_default_fp16_io,
649 states=states,
650 )
652 if exact_target == "mlprogram" and mlmodel._input_has_infinite_upper_bound():
653 raise ValueError(
654 "For mlprogram, inputs with infinite upper_bound is not allowed. Please set upper_bound"
655 ' to a positive value in "RangeDim()" for the "inputs" param in ct.convert().'
656 )
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/converter.py:186, in mil_convert(model, convert_from, convert_to, compute_units, **kwargs)
147 @_profile
148 def mil_convert(
149 model,
(...)
153 **kwargs
154 ):
155 """
156 Convert model from a specified frontend `convert_from` to a specified
157 converter backend `convert_to`.
(...)
184 See `coremltools.converters.convert`
185 """
--> 186 return _mil_convert(
187 model,
188 convert_from,
189 convert_to,
190 ConverterRegistry,
191 ct.models.MLModel,
192 compute_units,
193 **kwargs,
194 )
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/converter.py:218, in _mil_convert(model, convert_from, convert_to, registry, modelClass, compute_units, **kwargs)
215 weights_dir = _tempfile.TemporaryDirectory()
216 kwargs["weights_dir"] = weights_dir.name
--> 218 proto, mil_program = mil_convert_to_proto(
219 model,
220 convert_from,
221 convert_to,
222 registry,
223 **kwargs
224 )
226 _reset_conversion_state()
228 if convert_to == 'milinternal':
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/converter.py:302, in mil_convert_to_proto(model, convert_from, convert_to, converter_registry, main_pipeline, **kwargs)
299 if convert_to == 'milinternal':
300 return None, prog
--> 302 PassPipelineManager.apply_pipeline(prog, backend_pipeline)
304 prog._check_early_error_out_for_invalid_program()
306 backend_converter_type = converter_registry.backends.get(convert_to.lower())
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/mil/passes/pass_pipeline.py:495, in PassPipelineManager.apply_pipeline(prog, pass_pipeline)
491 except Exception as e:
492 logger.error(
493 f"\n\nERROR - '{pass_name}' graph pass produces the following error:\n"
494 )
--> 495 raise e # re-raise exception
497 # After dead code elimination, we should check if the program misses any essential scope info
498 check_essential_scope = pass_name == "common::dead_code_elimination"
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/mil/passes/pass_pipeline.py:490, in PassPipelineManager.apply_pipeline(prog, pass_pipeline)
487 graph_pass.set_options(pass_options)
489 try:
--> 490 graph_pass(prog)
491 except Exception as e:
492 logger.error(
493 f"\n\nERROR - '{pass_name}' graph pass produces the following error:\n"
494 )
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/mil/passes/graph_pass.py:56, in AbstractGraphPass.__call__(self, prog)
52 if not prog.skip_all_passes:
53 # we use the scope context manager to populate the graph pass information to the ops
54 # constructed by the pass.
55 with mb.scope(ScopeInfo(source=ScopeSource.COREMLTOOLS_GRAPH_PASS, data=[str(self)])):
---> 56 self.apply(prog)
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/backend/mil/passes/insert_image_preprocessing_op.py:29, in insert_image_preprocessing_ops.apply(self, prog)
27 # The new image processing ops will be inserted in front of the first op in the function
28 with mb.set_before_op(f.operations[0]):
---> 29 _insert_image_preprocessing_ops(f)
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/mil/passes/helper.py:64, in block_context_manager.<locals>.wrapper(*args)
58 raise ValueError(
59 "The function decorated with block_context_manager must have a Block "
60 "type argument as the first input."
61 )
63 with block:
---> 64 return _func(*args)
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/backend/mil/passes/insert_image_preprocessing_op.py:48, in _insert_image_preprocessing_ops(block)
46 input_nptype = nptype_from_builtin(type(last_output.dtype()))
47 if input_type.scale != 1:
---> 48 last_output = mb.mul(
49 x=last_output,
50 y=np.array(input_type.scale, dtype=input_nptype),
51 name=input_var.name + "__scaled__",
52 )
53 if has_bias:
54 if input_type.color_layout in (
55 _input_types.ColorLayout.GRAYSCALE,
56 _input_types.ColorLayout.GRAYSCALE_FLOAT16,
57 ):
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/mil/ops/registry.py:183, in SSAOpRegistry.register_op.<locals>.class_wrapper.<locals>.add_op(cls, **kwargs)
180 else:
181 op_cls_to_add = op_reg[op_type]
--> 183 return cls._add_op(op_cls_to_add, **kwargs)
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/mil/builder.py:237, in Builder._add_op(cls, op_cls, **kwargs)
235 curr_block()._insert_op_before(new_op, before_op=before_op)
236 new_op.build_nested_blocks()
--> 237 new_op.type_value_inference()
238 if len(new_op.outputs) == 1:
239 return new_op.outputs[0]
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/mil/operation.py:262, in Operation.type_value_inference(self, overwrite_output)
252 def type_value_inference(self, overwrite_output=False):
253 """
254 Perform type inference and auto_val computation based on new input Vars
255 in kwargs. If self._output_vars is None then we generate _output_vars;
(...)
260 existing _output_vars
261 """
--> 262 output_types = self.type_inference()
263 if not isinstance(output_types, tuple):
264 output_types = (output_types,)
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/mil/ops/defs/iOS15/elementwise_binary.py:45, in elementwise_binary.type_inference(self)
42 raise ValueError("Incompatible primitive types in broadcast operation")
43 primitive_type = self.get_dtype(primitive_type)
---> 45 return infer_type_with_broadcast(typea, typeb, primitive_type)
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/mil/ops/defs/_utils.py:126, in infer_type_with_broadcast(typea, typeb, primitive_type)
124 shapea = list(typea.get_shape())
125 shapeb = list(typeb.get_shape())
--> 126 ret_shape = broadcast_shapes(shapea, shapeb)
127 return types.tensor(primitive_type, ret_shape)
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/mil/ops/defs/_utils.py:92, in broadcast_shapes(shape_x, shape_y)
90 elif not is_y_unknown and shape_y[i] > 1:
91 if not is_x_unknown and shape_x[i] != shape_y[i]:
---> 92 raise_incompatible_dim_exception()
93 ret_shapes.append(shape_y[i])
94 elif not is_x_unknown and shape_x[i] > 1:
File ~/anaconda3/envs/tf-gpu-coreml/lib/python3.9/site-packages/coremltools/converters/mil/mil/ops/defs/_utils.py:66, in broadcast_shapes.<locals>.raise_incompatible_dim_exception()
65 def raise_incompatible_dim_exception():
---> 66 raise ValueError(
67 "Incompatible dim {} in shapes {} vs. {}".format(
68 i, shape_x, shape_y
69 )
70 )
ValueError: Incompatible dim 3 in shapes (1, 3, 224, 224) vs. (1, 1, 1, 3)
To Reproduce
import coremltools as ct
import tensorflow as tf
keras_model = tf.keras.applications.MobileNetV2()
input_type = ct.ImageType(shape=(1, 224, 224, 3), scale=[1/127.5]*3, bias=[-1, -1, -1], color_layout='RGB')
coreml_model = ct.convert(keras_model, inputs=[input_type])
System environment (please complete the following information):
- coremltools version: 8.2
- OS: Linux Ubuntu 22.04
- TensorFlow version: 2.12.0
Hey @SergeyMilyaev! Happy to hop on a call to try help you debug this. My email / Calendly is on my profile if you're interested :)
