core dump when giving Conv2DTranspose layer empty tensor and "valid" padding mode in GPU mode.

[maybeLee]

This issue is similar to my previous issue: https://github.com/keras-team/keras/issues/16933#issuecomment-1235897857 but the symptom and fault triggering condition are different.

In short, when I config Conv2DTranspose in "valid" padding mode and then give it an empty tensor, it will lead to a core dump on GPU. Different from the post: https://github.com/keras-team/keras/issues/16933#issuecomment-1235897857, it will also lead to a crash on CPU with the following error message:

File "/usr/local/lib/python3.7/dist-packages/keras/layers/convolutional.py", line 1343, in call
      dilation_rate=self.dilation_rate)
    File "/usr/local/lib/python3.7/dist-packages/keras/backend.py", line 5722, in conv2d_transpose
      data_format=tf_data_format)
Node: 'model_5/conv2d_transpose_6/conv2d_transpose'
Failed to call dnnl_sgemm. Error code: 2
	 [[{{node model_5/conv2d_transpose_6/conv2d_transpose}}]] [Op:__inference_predict_function_825]

To reproduce this issue, please run the following code on GPU:

import keras
input_shape = [None, 7, 0, 8]
x = keras.layers.Input(input_shape[1:])
layer = keras.layers.Conv2DTranspose(3, 3, strides=(1, 1), dtype="float32")
y = layer(x)
model = keras.models.Model(x,y)
model.summary()
import numpy as np
input_shape[0] = 10
test_input = np.random.rand(*input_shape)
res = model.predict(test_input)
print(res)

You can also access my colab: https://colab.research.google.com/drive/1cIlK9OcZJ6y9yIA3gR4nLgCFlFkVj4GV?usp=sharing Interestingly, I notice that if I configure Conv2DTranspose in "same" padding mode, it can work normally on both CPU and GPU. In contrast, Conv3DTranspose will lead to a core dump regardless of the padding mode as long as the input is an empty tensor and it is executed on GPU.

[sushreebarsa]

@maybeLee I tried to replicate the issue on colab using TF v2.9 and tf-nightly(2.11.0.dev20220906), but I didn’t face any issue as reported in tf-nightly. Could you please find the gist here and confirm the same? Thank you!

[maybeLee]

Hi @sushreebarsa , I confirm your observation. The core dump issue happens when I using TensorFlow 2.8.2, I tried on TensorFlow 2.9.0 and tf-nightly, there is no core dump. However, when I run the above program with tf-nightly on GPU, it works properly but it fails with the following error message on CPU.

Node: 'model/conv2d_transpose/conv2d_transpose'
Failed to call dnnl_sgemm. Error code: 2
	 [[{{node model/conv2d_transpose/conv2d_transpose}}]] [Op:__inference_predict_function_137]

Please check if it is a bug or it is an expected behavior

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No

[aliakdeniz38]

Unsubscribe

On Mon, Sep 5, 2022 at 9:33 AM maybeLee @.***> wrote:

This issue is similar to my previous issue: #16933 (comment) but the symptom and fault triggering condition are different.

In short, when I config Conv2DTranspose in "valid" padding mode and then give it an empty tensor, it will lead to a core dump on GPU. Different from the post: #16933 (comment), it will also lead to a crash on CPU with the following error message:

File "/usr/local/lib/python3.7/dist-packages/keras/layers/convolutional.py", line 1343, in call dilation_rate=self.dilation_rate) File "/usr/local/lib/python3.7/dist-packages/keras/backend.py", line 5722, in conv2d_transpose data_format=tf_data_format) Node: 'model_5/conv2d_transpose_6/conv2d_transpose' Failed to call dnnl_sgemm. Error code: 2 [[{{node model_5/conv2d_transpose_6/conv2d_transpose}}]] [Op:__inference_predict_function_825]

To reproduce this issue, please run the following code on GPU:

import keras input_shape = [None, 7, 0, 8] x = keras.layers.Input(input_shape[1:]) layer = keras.layers.Conv2DTranspose(3, 3, strides=(1, 1), dtype="float32") y = layer(x) model = keras.models.Model(x,y) model.summary() import numpy as np input_shape[0] = 10 test_input = np.random.rand(*input_shape) res = model.predict(test_input) print(res)

You can also access my colab: https://colab.research.google.com/drive/1cIlK9OcZJ6y9yIA3gR4nLgCFlFkVj4GV?usp=sharing Interestingly, I notice that if I configure Conv2DTranspose in "same" padding mode, it can work normally on both CPU and GPU. In contrast, Conv3DTranspose will lead to a core dump regardless of the padding mode as long as the input is an empty tensor and it is executed on GPU.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[gowthamkpr]

@maybeLee I don't run into any error on CPU with tf-nughtly. Please find the gist

[maybeLee]

@gowthamkpr ,

Thanks for your investigation. I tried on CPU with tf-nightly again, it seems that this issue has been fixed. Thanks for your help.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No