Error When Converting Keras Model to TensorFlow Lite Format

[lxzheng]

Environment:

• TensorFlow Version: 2.15.0.post1
• Keras Version: 3.0.4

Issue Description:

An error occurs when converting a Keras model to TensorFlow Lite format. This issue arises using tf.lite.TFLiteConverter.from_keras_model or tf.lite.TFLiteConverter.from_saved_model methods.

Steps to Reproduce using tf.lite.TFLiteConverter.from_keras_model:

Below is the code that reproduces this error:

import tensorflow as tf
import keras

(train_images, train_labels), (test_images, test_labels) = keras.datasets.fashion_mnist.load_data()

test_model = keras.Sequential([
    keras.layers.Flatten(),
    keras.layers.Dense(64, activation='relu'),
    keras.layers.Dense(10, activation='softmax')
])
test_model.compile(optimizer='adam',
              loss=keras.losses.SparseCategoricalCrossentropy(),
              metrics=['accuracy'])
test_model.fit(train_images, train_labels, epochs=1)

# Attempt to convert the model to TFLite format
converter = tf.lite.TFLiteConverter.from_keras_model(test_model)
tflite_model = converter.convert()

# Error also occurs when converting after saving the model as SavedModel format
tf.saved_model.save(test_model, "test")
converter = tf.lite.TFLiteConverter.from_saved_model("test")
tflite_model = converter.convert()

Error Message:

2024-01-26 16:09:46.247177: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:378] Ignored output_format. 2024-01-26 16:09:46.247219: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:381] Ignored drop_control_dependency. 2024-01-26 16:09:46.248043: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: /tmp/tmppf9ya5y2 2024-01-26 16:09:46.248474: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2024-01-26 16:09:46.248492: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: /tmp/tmppf9ya5y2 2024-01-26 16:09:46.251628: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:388] MLIR V1 optimization pass is not enabled 2024-01-26 16:09:46.252135: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2024-01-26 16:09:46.273872: I tensorflow/cc/saved_model/loader.cc:217] Running initialization op on SavedModel bundle at path: /tmp/tmppf9ya5y2 2024-01-26 16:09:46.280179: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 32137 microseconds. 2024-01-26 16:09:46.294610: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var MLIR_CRASH_REPRODUCER_DIRECTORY to enable. loc(fused["ReadVariableOp:", callsite("sequential_1/dense_1/add/ReadVariableOp@__inference_serving_default_1691"("/tf/keras4pi/fashion_mnist.py":55:1) at callsite("/usr/local/lib/python3.11/dist-packages/tensorflow/lite/python/lite.py":1139:1 at callsite("/usr/local/lib/python3.11/dist-packages/tensorflow/lite/python/lite.py":1093:1 at callsite("/usr/local/lib/python3.11/dist-packages/tensorflow/lite/python/lite.py":1601:1 at callsite("/usr/local/lib/python3.11/dist-packages/tensorflow/lite/python/lite.py":1579:1 at callsite("/usr/local/lib/python3.11/dist-packages/tensorflow/lite/python/convert_phase.py":205:1 at callsite("/usr/local/lib/python3.11/dist-packages/tensorflow/lite/python/lite.py":1502:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/backend/tensorflow/layer.py":57:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/backend/tensorflow/layer.py":119:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py":118:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/layers/layer.py":831:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py":118:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/ops/operation.py":42:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py":157:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/models/sequential.py":203:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/models/functional.py":188:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/ops/function.py":153:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/models/functional.py":572:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py":118:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/layers/layer.py":831:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py":118:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/ops/operation.py":42:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py":157:1 at callsite("/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py":139:1 at "/usr/local/lib/python3.11/dist-packages/keras/src/backend/tensorflow/core.py":65:1))))))))))))))))))))))))]): error: missing attribute 'value' LLVM ERROR: Failed to infer result type(s). Aborted (core dumped)

Steps to Reproduce using tf.lite.TFLiteConverter.from_saved_model:

Below is the code that reproduces this error:

import tensorflow as tf
import keras

(train_images, train_labels), (test_images, test_labels) = keras.datasets.fashion_mnist.load_data()

test_model = keras.Sequential([
    keras.layers.Flatten(),
    keras.layers.Dense(64, activation='relu'),
    keras.layers.Dense(10, activation='softmax')
])
test_model.compile(optimizer='adam',
              loss=keras.losses.SparseCategoricalCrossentropy(),
              metrics=['accuracy'])
test_model.fit(train_images, train_labels, epochs=1)

# Error also occurs when converting after saving the model as SavedModel format
tf.saved_model.save(test_model, "test")
converter = tf.lite.TFLiteConverter.from_saved_model("test")
tflite_model = converter.convert()

Error Message:

2024-01-26 16:36:52.478718: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:378] Ignored output_format. 2024-01-26 16:36:52.478757: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:381] Ignored drop_control_dependency. 2024-01-26 16:36:52.479612: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: test 2024-01-26 16:36:52.480166: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2024-01-26 16:36:52.480185: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: test 2024-01-26 16:36:52.483729: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:388] MLIR V1 optimization pass is not enabled 2024-01-26 16:36:52.484242: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2024-01-26 16:36:52.505957: I tensorflow/cc/saved_model/loader.cc:217] Running initialization op on SavedModel bundle at path: test 2024-01-26 16:36:52.512151: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 32542 microseconds. 2024-01-26 16:36:52.523977: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var MLIR_CRASH_REPRODUCER_DIRECTORY to enable. loc(fused["ReadVariableOp:", "sequential_1/dense_1/add/ReadVariableOp@__inference_serving_default_1450"]): error: missing attribute 'value' LLVM ERROR: Failed to infer result type(s). Aborted (core dumped)

releated issue #19100

[lxzheng]

I have tested the same code in an environment where TensorFlow 2.15.0.post1 is installed without Keras 3.0.4, with a slight modification: replacing import keras with from tensorflow import keras. In this setup, I found that the code works normally. This further suggests that the error might be specifically related to the Keras 3.0.4.

[james77777778]

Hi @lxzheng You should use the new API (keras.Model.export) introduced in Keras 3

A working example:

import tensorflow as tf

import keras

(train_images, train_labels), (
    test_images,
    test_labels,
) = keras.datasets.fashion_mnist.load_data()

test_model = keras.Sequential(
    [
        keras.layers.Flatten(),
        keras.layers.Dense(64, activation="relu"),
        keras.layers.Dense(10, activation="softmax"),
    ]
)
test_model.compile(
    optimizer="adam",
    loss=keras.losses.SparseCategoricalCrossentropy(),
    metrics=["accuracy"],
)
test_model.fit(train_images, train_labels, epochs=1)
test_model.export("test", "tf_saved_model")  # replace tf.saved_model.save with this line
converter = tf.lite.TFLiteConverter.from_saved_model("test")
tflite_model = converter.convert()
with open("model.tflite", "wb") as f:
    f.write(tflite_model)

If you are interested in onnx export using torch backend, you might want to check out my project here 😃: https://github.com/james77777778/keras-image-models/blob/main/kimm/export/export_onnx.py

[lxzheng]

Hi @james77777778 ,

Thank you for sharing your suggestion to use the keras.Model.export API! I tested it out, and it worked as you described.

I did notice a small potential bug in the code snippet you provided. The export method should be called with a single argument, the file path, rather than two arguments. The correct usage would be test_model.export("test").

I also had a question: I couldn't find the export API in the Keras 3.0 documentation (keras.io), even though it exists in the source code. Is this an undocumented API?

Thank you again for your helpful response and for sharing your project!

[SuryanarayanaY]

Hi @lxzheng ,

model.export is an inbuilt method of keras model. It has one default argument also format="tf_saved_model".

I also had a question: I couldn't find the export API in the Keras 3.0 documentation (keras.io), even though it exists in the source code. Is this an undocumented API?

The documentation part needs to be checked as most of these inbuilt methods documentation not generated.

[lxzheng]

Hi @SuryanarayanaY After reading the Keras source code, I noticed that the model.export function is actually implemented using ExportArchive, which ultimately calls tf.saved_model.save(). This raises a question: does saving a Keras 3.0 model in the SavedModel format require additional settings, or can we directly use tf.saved_model.save()?

I encountered errors when converting a model saved with tf.saved_model.save() to the TFLite format, although it seems to work fine with TensorFlow Serving. This discrepancy suggests that there might be specific requirements or steps that need to be followed when saving Keras 3.0 models for certain use cases, especially for TFLite conversion.

Could there be additional configurations or steps within model.export that are essential for ensuring compatibility with TFLite, which might be missing when directly using tf.saved_model.save()? Understanding this could be crucial for developers who need to deploy their models across various platforms and formats.

[SuryanarayanaY]

After reading the Keras source code, I noticed that the model.export function is actually implemented using ExportArchive, which ultimately calls tf.saved_model.save(). This raises a question: does saving a Keras 3.0 model in the SavedModel format require additional settings, or can we directly use tf.saved_model.save()?

Hi @lxzheng , As per migration documentation here saving a keras3 model into tf saved_model needs to use tf.saved_model.save API but not model.save. This is the only limitation AFAIK.

However for loading the tf saved_model we need to call the model on TFSM layer like below.

keras.layers.TFSMLayer("saved_model", call_endpoint="serving_default")

I encountered errors when converting a model saved with tf.saved_model.save() to the TFLite format, although it seems to work fine with TensorFlow Serving. This discrepancy suggests that there might be specific requirements or steps that need to be followed when saving Keras 3.0 models for certain use cases, especially for TFLite conversion.

Whether the TFlite conversion issue happens with Keras3 model only ? Pr ot happens with keras2 i,e tf.keras also ? Could you please submit a reproducible code snippet for same ?

Thanks!

[lxzheng]

Hi @SuryanarayanaY , Thank you for your response and for pointing out the migration documentation regarding the saving of Keras 3 models using the tf.saved_model.save API. I appreciate the clarification on the process and the introduction of the TFSMLayer for loading the saved model.

Whether the TFlite conversion issue happens with Keras3 model only ? Pr ot happens with keras2 i,e tf.keras also ? Could you please submit a reproducible code snippet for same ?

Yes, I have verified and mentioned in the issue that this bug occurs exclusively with Keras 3 and not with Keras 2 (i.e., tf.keras). Here's the relevant comment for reference:

I have tested the same code in an environment where TensorFlow 2.15.0.post1 is installed without Keras 3.0.4, with a slight modification: replacing import keras with from tensorflow import keras. In this setup, I found that the code works normally. This further suggests that the error might be specifically related to Keras 3.0.4.

Additionally, I had provided the source code in my initial issue description that demonstrates the error when converting a model saved with tf.saved_model.save() to TensorFlow Lite format.

import tensorflow as tf
import keras 

(train_images, train_labels), (test_images, test_labels) = keras.datasets.fashion_mnist.load_data()

test_model = keras.Sequential([
    keras.layers.Flatten(),
    keras.layers.Dense(64, activation='relu'),
    keras.layers.Dense(10, activation='softmax')
])
test_model.compile(optimizer='adam',
              loss=keras.losses.SparseCategoricalCrossentropy(),
              metrics=['accuracy'])
test_model.fit(train_images, train_labels, epochs=1)

# Error also occurs when converting after saving the model as SavedModel format
tf.saved_model.save(test_model, "test")
converter = tf.lite.TFLiteConverter.from_saved_model("test")
tflite_model = converter.convert()

Thanks!

[lxzheng]

Hi @SuryanarayanaY ,

I have another question. I found that using model.export or keras.export.ExportArchive() allows for successful saving of models with image augmentation layers, which addresses the problem mentioned in #19100 where tf.saved_model.save() fails. However, models saved using model.export() cannot be converted to TFLite format. In contrast, using keras.export.ExportArchive(), and specifying training=False in add_endpoint, does enable TFLite conversion. This leads me to question why models with augmentation layers, saved using model.export, cannot be converted to TFLite format. Do you have any insights on this?

Here is the code snippet that demonstrates conversion to TFLite format:

import tensorflow as tf
import keras

(train_images, train_labels), (test_images, test_labels) = keras.datasets.fashion_mnist.load_data()

model = keras.Sequential([
    keras.layers.RandomFlip("horizontal_and_vertical"),
    keras.layers.RandomRotation(0.1),
    
    keras.layers.Flatten(),
    keras.layers.Dense(64, activation='relu'),
    
    keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam',
              loss=keras.losses.SparseCategoricalCrossentropy(),
              metrics=['accuracy'])
model.fit(train_images, train_labels, epochs=1)

export_archive = keras.export.ExportArchive()
export_archive.track(model)
export_archive.add_endpoint(
    name="call_inference",
    fn=lambda x: model.call(x, training=False),
    input_signature=[tf.TensorSpec(shape=(None, 28, 28), dtype=tf.uint8)],
)
export_archive.write_out("test/2")

print("--convert from SavedModel by ExportArchive--")
converter = tf.lite.TFLiteConverter.from_saved_model("test/2")
tflite_model = converter.convert()
with open('test_model.tflite', 'wb') as f:
    f.write(tflite_model)

model.export("test/1")
print("--Error:convert from SavedModel by  model.export --")
converter=tf.lite.TFLiteConverter.from_saved_model("test/1")
tflite_model = converter.convert()
with open('test_model.tflite', 'wb') as f:
    f.write(tflite_model)

Error Message: Converting with model.export

2024-02-01 15:48:38.485862: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:378] Ignored output_format. 2024-02-01 15:48:38.485912: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:381] Ignored drop_control_dependency. 2024-02-01 15:48:38.486166: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: test/1 2024-02-01 15:48:38.487705: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2024-02-01 15:48:38.487730: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: test/1 2024-02-01 15:48:38.493220: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2024-02-01 15:48:38.599381: I tensorflow/cc/saved_model/loader.cc:217] Running initialization op on SavedModel bundle at path: test/1 2024-02-01 15:48:38.635194: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 149029 microseconds. error: 'tfl.assign_variable' op operand #1 must be tensor of 32-bit float or 64-bit float or 1-bit signless integer or 8-bit unsigned integer or 8-bit signless integer or QI8 type or QUI8 type or 32-bit signless integer or 64-bit signless integer or QI16 type or complex type with 32-bit float elements or complex type with 64-bit float elements values, but got 'tensor<2xui32>' error: 'tfl.assign_variable' op operand #1 must be tensor of 32-bit float or 64-bit float or 1-bit signless integer or 8-bit unsigned integer or 8-bit signless integer or QI8 type or QUI8 type or 32-bit signless integer or 64-bit signless integer or QI16 type or complex type with 32-bit float elements or complex type with 64-bit float elements values, but got 'tensor<2xui32>'

ConverterError Traceback (most recent call last) Cell In[13], line 9 7 print("--convert from SavedModel by model.export--") 8 converter=tf.lite.TFLiteConverter.from_saved_model("test/1") ----> 9 tflite_model = converter.convert() 10 with open('test_model.tflite', 'wb') as f: 11 f.write(tflite_model)

File /usr/local/lib/python3.11/dist-packages/tensorflow/lite/python/lite.py:1139, in _export_metrics..wrapper(self, *args, **kwargs) 1136 @functools.wraps(convert_func) 1137 def wrapper(self, *args, **kwargs): 1138 # pylint: disable=protected-access -> 1139 return self._convert_and_export_metrics(convert_func, *args, **kwargs)

File /usr/local/lib/python3.11/dist-packages/tensorflow/lite/python/lite.py:1093, in TFLiteConverterBase._convert_and_export_metrics(self, convert_func, *args, **kwargs) 1091 self._save_conversion_params_metric() 1092 start_time = time.process_time() -> 1093 result = convert_func(self, *args, **kwargs) 1094 elapsed_time_ms = (time.process_time() - start_time) * 1000 1095 if result:

File /usr/local/lib/python3.11/dist-packages/tensorflow/lite/python/lite.py:1465, in TFLiteSavedModelConverterV2.convert(self) 1459 else: 1460 self._debug_info = _get_debug_info( 1461 _convert_debug_info_func(self._trackable_obj.graph_debug_info), 1462 graph_def, 1463 ) -> 1465 return self._convert_from_saved_model(graph_def)

File /usr/local/lib/python3.11/dist-packages/tensorflow/lite/python/lite.py:1331, in TFLiteConverterBaseV2._convert_from_saved_model(self, graph_def) 1328 converter_kwargs.update(self._get_base_converter_args()) 1329 converter_kwargs.update(quant_mode.converter_flags()) -> 1331 result = _convert_saved_model(**converter_kwargs) 1332 return self._optimize_tflite_model( 1333 result, quant_mode, quant_io=self.experimental_new_quantizer 1334 )

File /usr/local/lib/python3.11/dist-packages/tensorflow/lite/python/convert_phase.py:212, in convert_phase..actual_decorator..wrapper(*args, **kwargs) 210 else: 211 report_error_message(str(converter_error)) --> 212 raise converter_error from None # Re-throws the exception. 213 except Exception as error: 214 report_error_message(str(error))

File /usr/local/lib/python3.11/dist-packages/tensorflow/lite/python/convert_phase.py:205, in convert_phase..actual_decorator..wrapper(*args, **kwargs) 202 @functools.wraps(func) 203 def wrapper(*args, **kwargs): 204 try: --> 205 return func(*args, **kwargs) 206 except ConverterError as converter_error: 207 if converter_error.errors:

File /usr/local/lib/python3.11/dist-packages/tensorflow/lite/python/convert.py:1001, in convert_saved_model(**kwargs) 999 model_flags = build_model_flags(**kwargs) 1000 conversion_flags = build_conversion_flags(**kwargs) -> 1001 data = convert( 1002 model_flags, 1003 conversion_flags, 1004 input_data_str=None, 1005 debug_info_str=None, 1006 enable_mlir_converter=True, 1007 ) 1008 return data

File /usr/local/lib/python3.11/dist-packages/tensorflow/lite/python/convert.py:366, in convert(model_flags, conversion_flags, input_data_str, debug_info_str, enable_mlir_converter) 358 conversion_flags.guarantee_all_funcs_one_use = True 359 return convert( 360 model_flags, 361 conversion_flags, (...) 364 enable_mlir_converter, 365 ) --> 366 raise converter_error 368 return _run_deprecated_conversion_binary( 369 model_flags.SerializeToString(), 370 conversion_flags.SerializeToString(), 371 input_data_str, 372 debug_info_str, 373 )

ConverterError: :0: error: 'tfl.assign_variable' op operand #1 must be tensor of 32-bit float or 64-bit float or 1-bit signless integer or 8-bit unsigned integer or 8-bit signless integer or QI8 type or QUI8 type or 32-bit signless integer or 64-bit signless integer or QI16 type or complex type with 32-bit float elements or complex type with 64-bit float elements values, but got 'tensor<2xui32>' :0: error: 'tfl.assign_variable' op operand #1 must be tensor of 32-bit float or 64-bit float or 1-bit signless integer or 8-bit unsigned integer or 8-bit signless integer or QI8 type or QUI8 type or 32-bit signless integer or 64-bit signless integer or QI16 type or complex type with 32-bit float elements or complex type with 64-bit float elements values, but got 'tensor<2xui32>'

[SuryanarayanaY]

Hi @lxzheng ,

I have replicated the reported error and attached gist here.

[elangoRamar]

Is there any update on this issue!

Because i am also facing the same issue. I am using tensorflow 2.16.2 and keras 3.5.0, And i am using tf.lite.TFLiteConverter.from_keras_model(model) for model conversion. The solution given in this comment works fine, but in my case, i have a custom layer, so i want to pass custom object when i load the model like below,

Converter

model = tf.keras.models.load_model(
		source, {'EntropyThresholdLayer': EntropyThresholdLayer})
	converter = tf.lite.TFLiteConverter.from_keras_model(model)
	tflite_model = converter.convert()
	with open(destination, 'wb') as f:
		f.write(tflite_model)

And i am using model.save('destination_path') to save the model.

is there any solution for this issue?