apple
Input error during converting torch.fft.irfft
🐞Describing the bug
I tried to convert the Pytorch model with the function torch.fft.irfft, but it failed with the error below: ValueError: Op "complex_irfft_0" (op_type: complex_irfft) Input data="k_f" expects tensor or scalar of dtype from type domain ['complex64'] but got tensor[3,fp32]
Stack Trace
scikit-learn version 1.3.1 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
Torch version 2.2.0.dev20231110+cu118 has not been tested with coremltools. You may run into unexpected errors. Torch 2.1.0 is the most recent version that has been tested.
tensor([ 2.5000+0.0000j, -0.6250+0.8602j, -0.6250+0.2031j],
dtype=torch.complex128)
=========testing irfft=====
tensor([0.1562, 0.3512, 0.7812, 1.2113], dtype=torch.float64)
=========test irfft in model====
tensor([0.1562, 0.3512, 0.7812, 1.2113], dtype=torch.float64)
========convert to coreml model=======
When both 'convert_to' and 'minimum_deployment_target' not specified, 'convert_to' is set to "mlprogram" and 'minimum_deployment_targer' is set to ct.target.iOS15 (which is same as ct.target.macOS12). Note: the model will not run on systems older than iOS15/macOS12/watchOS8/tvOS15. In order to make your model run on older system, please set the 'minimum_deployment_target' to iOS14/iOS13. Details please see the link: https://coremltools.readme.io/docs/unified-conversion-api#target-conversion-formats
Model is not in eval mode. Consider calling '.eval()' on your model prior to conversion
Converting PyTorch Frontend ==> MIL Ops: 75%|████████████████████████████████████████████████████████████████████▎ | 3/4 [00:00<00:00, 6872.15 ops/s]
Traceback (most recent call last):
File "/data/Model_convert/irfft_test/test_irfft_bug.py", line 50, in <module>
main()
File "/data/Model_convert/irfft_test/test_irfft_bug.py", line 40, in main
mlmodel = ct.convert(
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/_converters_entry.py", line 574, in convert
mlmodel = mil_convert(
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 188, in mil_convert
return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 212, in _mil_convert
proto, mil_program = mil_convert_to_proto(
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 286, in mil_convert_to_proto
prog = frontend_converter(model, **kwargs)
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 108, in __call__
return load(*args, **kwargs)
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 80, in load
return _perform_torch_convert(converter, debug)
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 99, in _perform_torch_convert
prog = converter.convert()
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 519, in convert
convert_nodes(self.context, self.graph)
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 88, in convert_nodes
add_op(context, node)
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 6179, in fft_irfft
irfft_res = mb.complex_irfft(data=input_data, n=n, dim=dim, norm=norm)
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/mil/mil/ops/registry.py", line 182, in add_op
return cls._add_op(op_cls_to_add, **kwargs)
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/mil/mil/builder.py", line 168, in _add_op
new_op = op_cls(**kwargs)
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/mil/mil/operation.py", line 190, in __init__
self._validate_and_set_inputs(input_kv)
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/mil/mil/operation.py", line 503, in _validate_and_set_inputs
self.input_spec.validate_inputs(self.name, self.op_type, input_kvs)
File "/home/yxh/anaconda3/envs/DSV2/lib/python3.9/site-packages/coremltools/converters/mil/mil/input_type.py", line 163, in validate_inputs
raise ValueError(msg.format(name, var.name, input_type.type_str,
ValueError: Op "complex_irfft_0" (op_type: complex_irfft) Input data="k_f" expects tensor or scalar of dtype from type domain ['complex64'] but got tensor[3,fp32]
To Reproduce
import torch
import torch.nn as nn
import numpy as np
import coremltools as ct
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
def forward(self, k_f):
k = torch.fft.irfft(k_f)
return k
def main():
k_f = np.array( [ 2.5000+0.0000j, -0.6250+0.8602j, -0.6250+0.2031j]) #np.load('k_f.npy')
k_f = torch.from_numpy(k_f)
print (k_f)
print ("=========testing irfft=====")
k_test = torch.fft.irfft(k_f)
print (k_test)
print ("=========test irfft in model====")
model = Net().to(torch.device("cpu"))
output = model(k_f)
print (output)
print ("========convert to coreml model=======")
trace = torch.jit.trace(model, (k_f))
#trace = torch.jit.script(model, (k_f))
input_k_f = ct.TensorType(
name='k_f',
shape=k_f.shape
)
mlmodel = ct.convert(
trace,
inputs=[input_k_f],
)
mlmodel.save("model.mlpackage")
print ("Coreml model saved...")
if __name__ == '__main__':
main()
System environment (please complete the following information):
- coremltools version: 7.1
- OS (e.g. MacOS version or Linux type): Ubuntu 20.04.4 LTS
- Any other relevant version information (e.g. PyTorch or TensorFlow version): Pytorch 1.13.1+cu116
Additional context
- Add anything else about the problem here that you want to share.
Thanks for the code. I can reproduce this issue.
I suspect the root issue here is that the Core ML Framework does not support complex number inputs. So this may not be possible to fix with the current Core ML Framework. However we should at least have a better error message.
Thanks for confirming the bug.
I found there is a workaround in the coremltools source code to support torch.fft.irfft in the Core ML Framework. coremltools/converters/mil/mil/ops/defs/complex_dialect_ops.py
The function name is complex_irfft in the source code.
Does that mean complex_irfft work for some use cases?
Thanks!
Coremltools does offer some partial support for complex numbers, but I believe we can not support models which input/output complex numbers.
Also in order to support a PyTorch ops we must implement it using our MIL ops. Given our current set of MIL ops, I don't think there is any reasonable way to implement fast Fourier transform related operations.
I don't think torch.fft.irfft is an operation we'll be able to support any time soon.