tfjs converter supported `tf.keras.layers.Conv2DTranspose` ?

[kaka-lin]

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

• Have I written custom code: No
• OS Platform and Distribution: MacOS
• TensorFlow.js installed from (npm or script link): npm
• TensorFlow.js version (use command below): 4.16.0
• Tensorflow.js Converter Version: 4.16.0

Describe the current behavior

My model has a layer as below:

tf.keras.layers.Conv2DTranspose(
    self.hidden_size // 4, kernel_size=2, strides=2, name="0.0", data_format="channels_first")

I converted this model to tfjs format and ran it, encountered an error as below:

util_base.ts:153 Uncaught (in promise) Error: Error in conv2dDerInput: depth of input (128) must match input depth for filter 64.
    at Module.assert (util_base.ts:153:1)
    at conv2DBackpropInput_ (conv2d_backprop_input.ts:87:1)
    at conv2DBackpropInput__op (operation.ts:51:1)
    at conv2dTranspose_ (conv2d_transpose.ts:54:1)
    at Module.conv2dTranspose__op (operation.ts:51:1)
    at Module.executeOp (convolution_executor.ts:183:1)
    at operation_executor.ts:69:1
    at engine.ts:469:1
    at Engine.scopedRun (engine.ts:480:1)
    at Engine.tidy (engine.ts:467:1)

And then when I edit conv2d_transpose.js

return conv2DBackpropInput(outputShape, $x, $filter, strides, pad, 'NHWC', dimRoundingMode);

to

return conv2DBackpropInput(outputShape, $x, $filter, strides, pad, 'NCHW', dimRoundingMode);

then occur an error like below:

Describe the expected behavior

Should be run without any error.

[gaikwadrahul8]

Hi, @kaka-lin

Thank you for bringing this issue to our attention and if possible could you please help me with your model and tfjs_converter command which you're using to convert your model into Tensorflow.js model so I'll try to replicate same behavior from my end also ?

If you're using Google colab to convert your model to Tensorflow.js please help me with that Google colab notebbok.

Thank you for your cooperation and patience.

[kaka-lin]

Hi, @gaikwadrahul8

My tfjs_converter command is:

tensorflowjs_converter \
    --input_format=tf_saved_model \
    --output_format=tfjs_graph_model \
    --signature_name=serving_default \
    --saved_model_tags=serve \
    weights/saved_model \
    weights/web_model

My model is modified by transformers/src/transformers/models/sam/modeling_tf_sam.py and saved to SaveModel format.

[kaka-lin]

Hi, @gaikwadrahul8

This repo: kaka-lin/EfficientSAM-tf2-demo includes my weights and related inference code, please check it, thank you!

[gaikwadrahul8]

Hi, @kaka-lin

I apologize for the delayed response and I see official documentation of tfjs_converter supported ops for conv2dTranspose Op and it's showing not mapped with Tensorflow Op so there's no direct equivalent operation in the TensorFlow backend at this time.

As far my current understanding you can't directly use that API in a TFJs model that needs to be executed by the TensorFlow backend so If you have the expertise create a custom layer that implements transposed convolution using supported TensorFlow ops so please feel free to do that.

Thank you for your cooperation and patience.

[kaka-lin]

Hi, @gaikwadrahul8

I created a custom layer of transposed convolution and found the problem is biasAdd does not support NCHW, so would occur the error below:

Also the same as issue #3282.

So I created a custom layer and changed NCHW to NHWC and it works, thank you!

def CustomConv2DTranspose(layer):
...
    def call(self, inputs): 
        ...
        output = tf.nn.conv2d_transpose(
                    inputs,
                    self.kernel,
                    output_shape=output_shape_tensor,
                    strides=self.strides,
                    padding=self.padding,
                )
        
        if self.use_bias:
            output = tf.nn.bias_add(
                output,
                self.bias,
                data_format="NHWC",
            )

So, TFJS Converter supported conv2d_transpose but does not support NCHW format? even if it has this arg and supports both NHWC and NCHW?