Relu activation in TF Lite model return the negative values

[luanft]

1. System information

• OS Platform and Distribution (Centos 7):
• TensorFlow installation (pip package):
• TensorFlow library (tensorflow==2.15.0):

2. Code

import tensorflow as tf
from tensorflow.keras.applications import ResNet50#, MobileNetV2
from tensorflow.keras.applications.resnet50 import preprocess_input
import tensorflow_datasets as tfds

resnet50 = ResNet50(
    include_top=True,
    weights='imagenet',
    input_tensor=None,
    input_shape=(224, 224, 3),
    pooling=None,
    classes=1000,
    classifier_activation='softmax'
)
resnet50.summary()

imagenet_v2 = tfds.load('imagenet_v2', split='test', as_supervised=True, shuffle_files=True,
                   download_and_prepare_kwargs={
                       'download_config': tfds.download.DownloadConfig(verify_ssl = False)
                   })

def representative_dataset():
    dataset = imagenet_v2.shuffle(1000).batch(1).take(200)
    for data, _ in dataset:
        image = tf.image.resize(data, [224, 224])
        yield [image]


converter = tf.lite.TFLiteConverter.from_keras_model(resnet50)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [
    tf.lite.OpsSet.TFLITE_BUILTINS_INT8,
    tf.lite.OpsSet.SELECT_TF_OPS
]
converter.target_spec.supported_types = [tf.int8]
converter.inference_input_type = tf.int8  # or tf.uint8
converter.inference_output_type = tf.int8  # or tf.uint8
# converter.target_spec.supported_types = [tf.float16]
uint8_quantized_model = converter.convert()

interpreter = tf.lite.Interpreter(model_content=uint8_quantized_model, experimental_preserve_all_tensors=True)
interpreter.allocate_tensors()
dataset = imagenet_v2.shuffle(1000).batch(1).take(10)

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

for img, label in dataset:
    img = tf.image.resize(img, (224, 224))
    img = preprocess_input(img)
    img = tf.cast(img, tf.int8)

    interpreter.set_tensor(input_details[0]['index'], img)
    interpreter.invoke()
    output = interpreter.get_tensor(output_details[0]['index'])

    print(interpreter.get_tensor(113).flatten())

3. Failure after conversion

I'm trying to extract the output of an int8 quantized model to debug model accuracy. I just realized that that Conv2d(relu) output contains a lot of negative values. I know that the Relu activation is fused to conv2d. Am I wrong or the tflite model has a bug?

This is a visualized model:

This is the script output:

[sushreebarsa]

@luanft

Please make sure the conversion is done correctly;

# Convert the model to TFLite
converter = tf.lite.TFLiteConverter.from_keras_model(nn_model)
tflite_model = converter.convert()

# Save the TFLite model
with open('model.tflite', 'wb') as f:
    f.write(tflite_model)

Kindly ensure that there is no post-processing step that alters the output values, especially turning positive values into negatives.

Thank you!

[luanft]

@sushreebarsa Yes, the conversion is done correctly. Also, I didn't modify the tensor output. So, it cannot change the output. Please see my attached run log and script. sample_dir.tar.gz

###################################################################
Converting to tf lite model...
/home/users/luant/nn_models_env/lib/python3.10/site-packages/tensorflow/lite/python/convert.py:953: UserWarning: Statistics for quantized inputs were expected, but not specified; continuing anyway.
  warnings.warn(
2024-06-28 14:54:56.289495: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:378] Ignored output_format.
2024-06-28 14:54:56.289531: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:381] Ignored drop_control_dependency.
2024-06-28 14:54:56.290169: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: /tmp/tmpk0b4s6xx
2024-06-28 14:54:56.310480: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve }
2024-06-28 14:54:56.310523: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: /tmp/tmpk0b4s6xx
2024-06-28 14:54:56.350321: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:388] MLIR V1 optimization pass is not enabled
2024-06-28 14:54:56.367995: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle.
2024-06-28 14:54:56.906898: I tensorflow/cc/saved_model/loader.cc:217] Running initialization op on SavedModel bundle at path: /tmp/tmpk0b4s6xx
2024-06-28 14:54:57.075488: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 785327 microseconds.
2024-06-28 14:54:57.253047: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
Summary on the non-converted ops:
---------------------------------
 * Accepted dialects: tfl, builtin, func
 * Non-Converted Ops: 111, Total Ops 189, % non-converted = 58.73 %
 * 111 ARITH ops

- arith.constant:  111 occurrences  (f32: 108, i32: 3)



  (f32: 16)
  (f32: 53)
  (f32: 1)
  (f32: 1)
  (f32: 1)
  (f32: 2)
  (f32: 1)
fully_quantize: 0, inference_type: 6, input_inference_type: INT8, output_inference_type: INT8
###################################################################

[sawantkumar]

Hi @luanft

This negative relu outputs that you mentioned are from the quantized model , but if you dequantize the relu outputs back to float, they will be non-negative values. To verify this just follow the below formula " actual_value = 0.12550510466098785 * (quantized_value + 128) " . Please let me know if you face any problems using the above approach.

[luanft]

Hi @sawantkumar,

How to calculate the value of constant 0.12550510466098785 in your formula? actual_value = 0.12550510466098785 * (quantized_value + 128)

How the converter picks up the range for activation quantization?

1. float64[-inf, 0, +inf] to [-128, 0, 127]
2. float64[0, +inf] to [-128, 0, 127]

[sawantkumar]

Hi @luanft ,

The value of the constant can been seen from the netron screenshot of the conv2d layer . And for calculating the range for conversion from float to int you can look at this

[luanft]

@sawantkumar Thank you. I got the issue. No problems with RELU activation. Let me close the bug.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No