RuntimeError: pytorch_half_pixel mode unsupported

[johannesSX]

Hello,

I want to convert a model from onnx to pb format in tensorflow. The following error occurs here:

File "/Users/.../convert_model.py", line 72, in <module>
    convert_onnx2pb()
  File "/Users/.../convert_model.py", line 44, in convert_onnx2pb
    tf_rep = prepare(onnx_model, strict=False, logging_level='DEBUG')
  File "/Users/.../opt/anaconda3/envs/big_convert_3/lib/python3.6/site-packages/onnx_tf-1.5.0-py3.6.egg/onnx_tf/backend.py", line 65, in prepare
    return cls.onnx_model_to_tensorflow_rep(model, strict)
  File "/Users/.../opt/anaconda3/envs/big_convert_3/lib/python3.6/site-packages/onnx_tf-1.5.0-py3.6.egg/onnx_tf/backend.py", line 85, in onnx_model_to_tensorflow_rep
    return cls._onnx_graph_to_tensorflow_rep(model.graph, opset_import, strict)
  File "/Users/.../opt/anaconda3/envs/big_convert_3/lib/python3.6/site-packages/onnx_tf-1.5.0-py3.6.egg/onnx_tf/backend.py", line 143, in _onnx_graph_to_tensorflow_rep
    onnx_node, tensor_dict, handlers, opset=opset, strict=strict)
  File "/Users/.../opt/anaconda3/envs/big_convert_3/lib/python3.6/site-packages/onnx_tf-1.5.0-py3.6.egg/onnx_tf/backend.py", line 245, in _onnx_node_to_tensorflow_op
    return handler.handle(node, tensor_dict=tensor_dict, strict=strict)
  File "/Users/.../opt/anaconda3/envs/big_convert_3/lib/python3.6/site-packages/onnx_tf-1.5.0-py3.6.egg/onnx_tf/handlers/handler.py", line 60, in handle
    cls.args_check(node, **kwargs)
  File "/Users/.../opt/anaconda3/envs/big_convert_3/lib/python3.6/site-packages/onnx_tf-1.5.0-py3.6.egg/onnx_tf/handlers/backend/resize.py", line 50, in args_check
    coordinate_transformation_mode, "Tensorflow")
  File "/Users/.../opt/anaconda3/envs/big_convert_3/lib/python3.6/site-packages/onnx_tf-1.5.0-py3.6.egg/onnx_tf/common/exception.py", line 50, in __call__
    raise self._func(self.get_message(op, framework))
RuntimeError: Resize coordinate_transformation_mode=pytorch_half_pixel is not supported in Tensorflow.

When executing the following code (prepare function):

from onnx_tf.backend import prepare
tf_rep = prepare(onnx_model, logging_level='DEBUG')

I use the follwing software/python packages:

• python 3.6.10
• TensorFlow 2.2.0
• onnx 1.7.0
• onnx-tf 1.5.0, but the resize11 branch from @winnietsang
  • if i use the master branch, the resize error mentioned here occurs. thats why i use the resize11 branch. Thank you @winnietsang for implementing this!
• onnx model have opset_version 11

How can i solve the coordinate_transformation_mode=pytorch_half_pixel problem? Any help would be greatly appreciated :)

[winnietsang]

@johannesSX Currently only limited modes/options introduced in Resize opset 11 are supported. More investigation needed for other modes/options due to framework incompatibility.

[jf99]

@johannesSX Currently only limited modes/options introduced in Resize opset 11 are supported.

~~Can you say which modes/options exactly are currently supported? Maybe I can modify my ONNX file to match onnx-tf's requirements.~~

Edit: Sorry, I was blind. The error message is quite clear and alternative coordinate_transformation_modes can be found in the ONNX doc.

[winnietsang]

@jf99 Resize required 4D input in Tensorflow. For opset 11, only the following attributes and inputs combination are supported in onnx-tf:

mode=nearest, coordinate_transformation_mode=align_corners, nearest_mode=round_prefer_ceil, can use scales(*) or sizes.
mode=nearest, coordinate_transformation_mode=asymmetric, nearest_mode=floor, can use scales(*) or sizes.
mode=nearest, coordinate_transformation_mode=tf_half_pixel_for_nn, nearest_mode=floor, can use scales(*) or sizes.
mode=linear, coordinate_transformation_mode=align_corners, can use scales(*) or sizes.
mode=linear, coordinate_transformation_mode=asymmetric, can use scales(*) or sizes.
mode=linear, coordinate_transformation_mode=half_pixel, can use scales(*) or sizes.
mode=cubic, coordinate_transformation_mode=align_corners, cubic_coeff_a=-0.5, exclude_outside=1, can use scales(*) or sizes.
mode=cubic, coordinate_transformation_mode=asymmetric, cubic_coeff_a=-0.5, exclude_outside=1, can use scales(*) or sizes.
mode=cubic, coordinate_transformation_mode=half_pixel, cubic_coeff_a=-0.5, exclude_outside=1, can use scales(*) or sizes.
mode=nearest, coordinate_transformation_mode=tf_crop_and_resize, extrapolation_value=any_float_value, nearest_mode=round_prefer_ceil, can use scales or sizes.
mode=linear, coordinate_transformation_mode=tf_crop_and_resize, extrapolation_value=any_float_value, can use scales or sizes.

Note (*): The accuracy of your model will go down, if the height and the width of the new sizes(scales * origial sizes) are not in whole numbers.

[Tian14267]

Do you solve your problem? I get same problem, and don't know how to do ..... @johannesSX

[johannesSX]

Do you solve your problem? I get same problem, and don't know how to do ..... @johannesSX

I tried to solve it, but it is still not working. I implement a resize version, where the error not occurred. But the generated results are still different.

My goal was, to convert an pth pytorch model to tensorflow. Because i had access to the underlying pytorch model it was possible to rebuild it manuelly in tensorflow. Then i wrote an programm, that transfered the weights directly from pth file to tensorflow pb file. And it is working.

That may also work for you, but is very time intensive.

[Tian14267]

Oh My god. It seems too complicated .I still don't know why we get 'pytorch_half_pixel' in onnx model. It seems happened in upsample in torch to onnx.
My head goes blank....

[zshn25]

Using align_corners=True in Upsample solved this for me

[cenkbircanoglu]

+1

[omerferhatt]

Please check https://github.com/onnx/onnx/issues/2784#issuecomment-633909328, setting opset_version=10 will solve the issue

[anilsathyan7]

When i tried setting opset_version=10 there was a warning during conversion process:

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:3103: UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details. warnings.warn("The default behavior for interpolate/upsample with float scale_factor changed " /usr/local/lib/python3.6/dist-packages/torch/onnx/symbolic_helper.py:267: UserWarning: You are trying to export the model with onnx:Resize for ONNX opset version 10. This operator might cause results to not match the expected results by PyTorch. ONNX's Upsample/Resize operator did not match Pytorch's Interpolation until opset 11. Attributes to determine how to transform the input were added in onnx:Resize in opset 11 to support Pytorch's behavior (like coordinate_transformation_mode and nearest_mode). We recommend using opset 11 and above for models using this operator.

The model was converted successfully; but the output from the onnx model was different from the original pytorch model.

The original model was trained with bilinear upsample and align corners=False F.interpolate(..., scale_factor=2, mode='bilinear', align_corners=False)

Even if we change align_corners=True in the model definition(and load weights), the output is different from the original trained model. How can we convert such a model to tensorflow, with correct upsample behaviour?

The only option seems to be to retrain the original model with align_corners=True and mode='bilinear'

[Adhithya-tech]

@zshn25 @anilsathyan7 Could you tell me how o change the model definition for pretrained models?

[mohammedayub44]

Hi, I'm doing text boundary detections using CRAFT and I get the same error as reported but only when I use opset_version=11 and then try to run the exported model in TensorFlow. When I don't give any version, I do get the warning but my model seems to run fine. I tested with couple of images and boundaries come out to be fairly okay.

I visualized both the models (with and without version 11) in Netron UpSample does get converted to Resize Op when using version 11. As reported by @winnietsang bilinear and half_pixel combination is not supported.

Let me know if there is a new version that supports bilinear upsample. Unfortunately cannot retrain the model, sticking to using the okayish one for now.

[mmeendez8]

Is there any new about this? Followed @omerferhatt suggestion of opset_version=10 but I get an error:

return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
Traceback (most recent call last):
  File "pytorch2onnx_seg.py", line 383, in <module>
    pytorch2onnx(
  File "pytorch2onnx_seg.py", line 190, in pytorch2onnx
    torch.onnx.export(
  File "/opt/miniconda3/envs/model_exporter/lib/python3.8/site-packages/torch/onnx/__init__.py", line 275, in export
    return utils.export(model, args, f, export_params, verbose, training,
  File "/opt/miniconda3/envs/model_exporter/lib/python3.8/site-packages/torch/onnx/utils.py", line 88, in export
    _export(model, args, f, export_params, verbose, training, input_names, output_names,
  File "/opt/miniconda3/envs/model_exporter/lib/python3.8/site-packages/torch/onnx/utils.py", line 718, in _export
    _check_onnx_proto(proto)
RuntimeError: Node (Resize_145) has input size 4 not in range [min=2, max=2].

==> Context: Bad node spec: input: "572" input: "582" input: "590" input: "589" output: "591" name: "Resize_145" op_type: "Resize" attribute { name: "coordinate_transformation_mode" s: "asymmetric" type: STRING } attribute { name: "cubic_coeff_a" f: -0.75 type: FLOAT } attribute { name: "mode" s: "nearest" type: STRING } attribute { name: "nearest_mode" s: "floor" type: STRING }

Also tried with @anilsathyan7 idea of setting align_corners=True andmode='bilinear' but no success.

This is the script I am using for converting the model: https://github.com/open-mmlab/mmsegmentation/blob/master/tools/pytorch2onnx.py

[harley-hu]

When i tried setting opset_version=10 there was a warning during conversion process:

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:3103: UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details. warnings.warn("The default behavior for interpolate/upsample with float scale_factor changed " /usr/local/lib/python3.6/dist-packages/torch/onnx/symbolic_helper.py:267: UserWarning: You are trying to export the model with onnx:Resize for ONNX opset version 10. This operator might cause results to not match the expected results by PyTorch. ONNX's Upsample/Resize operator did not match Pytorch's Interpolation until opset 11. Attributes to determine how to transform the input were added in onnx:Resize in opset 11 to support Pytorch's behavior (like coordinate_transformation_mode and nearest_mode). We recommend using opset 11 and above for models using this operator.

The model was converted successfully; but the output from the onnx model was different from the original pytorch model.

The original model was trained with bilinear upsample and align corners=False F.interpolate(..., scale_factor=2, mode='bilinear', align_corners=False)

Even if we change align_corners=True in the model definition(and load weights), the output is different from the original trained model. How can we convert such a model to tensorflow, with correct upsample behaviour?

The only option seems to be to retrain the original model with align_corners=True and mode='bilinear'

@anilsathyan7 Yes，I also found it. But seem not to find better ideas to solve it.

[MohamedAliRashad]

The problem happened with me, any solutions ?

[PatShot]

This issue still persists while converting from an onnx converted from a PyTorch model, converted using torch.onnx.export module.

`RuntimeError: in user code:

File "/home/ubuntu/model_conv/.venv/lib/python3.8/site-packages/onnx_tf/backend_tf_module.py", line 99, in __call__  *
    output_ops = self.backend._onnx_node_to_tensorflow_op(onnx_node,
File "/home/ubuntu/model_conv/.venv/lib/python3.8/site-packages/onnx_tf/backend.py", line 347, in _onnx_node_to_tensorflow_op  *
    return handler.handle(node, tensor_dict=tensor_dict, strict=strict)
File "/home/ubuntu/model_conv/.venv/lib/python3.8/site-packages/onnx_tf/handlers/handler.py", line 58, in handle  *
    cls.args_check(node, **kwargs)
File "/home/ubuntu/model_conv/.venv/lib/python3.8/site-packages/onnx_tf/handlers/backend/resize.py", line 125, in args_check  *
    exception.OP_UNSUPPORTED_EXCEPT(
File "/home/ubuntu/model_conv/.venv/lib/python3.8/site-packages/onnx_tf/common/exception.py", line 50, in __call__  *
    raise self._func(self.get_message(op, framework))

RuntimeError: Resize coordinate_transformation_mode=pytorch_half_pixel is not supported in Tensorflow.`

The following Python packages were used in the model:

• torch 1.10.2
• torchvision 0.11.3

I used:

• onnx-tf == 1.10.0
• tensorflow == 2.8.0
• tensorflow-addons == 0.16.1
• tensorflow-probability == 0.16.0
• Python == 3.8.13

Onnx opset_version = 12 I have tried exporting it by changing it to various values between 9 and 12.

The idea was to convert from PyTorch model to a tf model via onnx. Any help is greatly appreciated.

[FabioRomagnolo]

Still getting this error like the others reported. This is supposedly caused by incompatibility with PyTorch's function for the interpolation.

Any news about this error?

[MohamedAliRashad]

Is there no hope for this issue to be solved ?

[FabioRomagnolo]

Not yet. If someone is interested I've written a script to convert the Resize nodes to 'half_pixel' mode, preventing the error due to 'pytorch_half_pixel' mode, still unsupported. Results will be obviously different at pixel level from the ones obtained by PyTorch, but at the end they will be similar at the perceptual visive level.

[MohamedAliRashad]

@FabioRomagnolo I would be happy to see this script

[FabioRomagnolo]

Ok, this is the code from my Github repository:

def save_new_model(opset_version, nodes, graph, out_path, verbose=True):
    if verbose:
        print("Creating new fixed graph...")
    # * create a new graph with new nodes.
    new_graph = h.make_graph(
        nodes,
        graph.name,
        graph.input,
        graph.output,
        initializer=graph.initializer,  # The initializer holds all non-constant weights.
    )
    if verbose:
        print("Creating new fixed model...")
    new_model = h.make_model(new_graph, producer_name="onnx-fix-nodes")
    new_model.opset_import[0].version = opset_version
    ch.check_model(new_model)
    if verbose:
        print(f"Saving new model as: {out_path}")
    onnx.save_model(new_model, out_path)


def fix_onnx_resize_nodes(model_path: str, out_path: str, verbose=True):
    """
    Method to fix resize nodes giving the following error in Tensorflow
    conversions:
    - "Resize coordinate_transformation_mode=pytorch_half_pixel is not supported in Tensorflow"
    """
    if verbose:
        print(f"Loading Model: {model_path}")
    # * load model.
    model = onnx.load_model(model_path)
    ch.check_model(model)
    # * get model opset version.
    opset_version = model.opset_import[0].version
    graph = model.graph

    new_nodes = []
    if verbose:
        print("Fixing Resize nodes...")
    for i, node in enumerate(graph.node):
        if node.op_type == "Resize":
            new_resize = onnx.helper.make_node(
                'Resize',
                inputs=node.input,
                outputs=node.output,
                name=node.name,
                coordinate_transformation_mode='half_pixel',  # Instead of pytorch_half_pixel, unsupported by Tensorflow
                mode='linear',
            )
            # Update node
            new_nodes += [new_resize]
        else:
            new_nodes += [node]

    save_new_model(opset_version=opset_version, graph=graph, nodes=new_nodes,
                   out_path=out_path, verbose=verbose)

If you're interested in further conversions like from 64 bits params to 32 bits params, the file to look in the repository is onnx_utils.py.

Hope this will help you!

UPDATE: I had to make the repository private. If you still need the other functions from onnx_utils.py I will share with you :)

[ivyas21]

@FabioRomagnolo,

Could you please share your onnx_utils.py code?

[Jwy-jump]

@FabioRomagnolo,

Could you please share your onnx_utils.py code?

[FabioRomagnolo]

Hi everyone! Sorry to read this late, @ivyas21 @Jwy-jump.

I created a new public repository with just the onnx_utils.py file. Take any method you need from the script. 🙂