ReLU layer does not clip the tensor into the desired range when put behind a convolution layer

[maybeLee]

Describe the bug When converting a keras model with the following architecture:

  keras.layers.Conv2D(filters=3, kernel_size=[3,3], strides=[2,2]),
  keras.layers.Lambda(lambda x: x * np.Inf),
  keras.layers.ReLU(max_value=6.0, negative_slope=0.0, threshold=0.0)

In keras, the tensor sent to ReLU will be clipped into [0.0, 6.0] and output something like this:

[[[[0. 0. 6.]
   [0. 0. 6.]]
  [[0. 0. 0.]
   [0. 0. 6.]]]
 [[[0. 0. 0.]
   [0. 0. 0.]]
  [[0. 6. 6.]
   [0. 6. 6.]]]
 [[[0. 6. 6.]
   [0. 0. 6.]]
  [[0. 0. 6.]
   [0. 6. 0.]]]
 [[[0. 0. 0.]
   [0. 6. 6.]]
  [[0. 0. 6.]
   [0. 0. 6.]]]

However, after this model is converted to ONNX, onnx will predict all NaN tensors. It seems like the parameter in ReLU layer: max_value and negative_slope do not work after converted to ONNX.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
• Tensorflow Version: 2.8.0
• Python version: 3.7.12

To Reproduce

# Create keras model
import keras
import numpy as np
input_shape = (10, 6, 6, 3)

x = keras.layers.Input(input_shape[1:])
layer_stack = [
  keras.layers.Conv2D(filters=3, kernel_size=[3,3], strides=[2,2]),
  keras.layers.Lambda(lambda x: x * np.Inf),
  keras.layers.ReLU(max_value=6.0, negative_slope=0.0, threshold=0.0)
]
layer_input = x
for layer in layer_stack:
  y = layer(layer_input)
  layer_input = y

model = keras.Model(x,y)
model.summary()

import numpy as np
x = np.random.rand(*input_shape)
res = model.predict(x)
print(res, res.shape)

# Convert to ONNX
import tensorflow as tf
import tf2onnx
input_shape = model.layers[0].input_shape[0]
spec = (tf.TensorSpec(input_shape, tf.float32, name="input"),)
model, _ = tf2onnx.convert.from_keras(model, input_signature=spec, \
        opset=15, output_path="temp")

# Conduct ONNX inference
import onnxruntime as ort
session = ort.InferenceSession("temp")
input_name = session.get_inputs()[0].name
output_name = session.get_outputs()[0].name
x = x.astype("float32")
result = session.run([output_name], {input_name: x})
print(result)

You may directly access this colab link for reproduction: https://colab.research.google.com/drive/1qewVbeYcLyilFya2cfHRS33rWGEfOMYx?usp=sharing

Additional context Interestingly, I find out that: if we remove the first architecture (i.e., keras.layers.Conv2D). ONNX will successfully clip the tensor into range [0,6] like Keras does.

Can you help check this issue?

[fatcat-z]

Tf2onnx uses a frozen graph to do the conversion. While tensorflow generates this frozen graph, it will change the inputs according to max_value and negative_slope accordingly. So the ONNX graph doesn't know the value of max_value and negative_slope and onnx op RELU doesn't support these as attributes.

[fatcat-z]

Closing this issue since it is a tensorflow issue. Feel free to open a new one if the issue still exists.