The tutorial Keras SavedModels from TF 2.7 do not load in TF 2.8, 2.9, 2.10

[kennysong]

System information.

• Have I written custom code (as opposed to using a stock example script provided in Keras): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Debian Buster
• TensorFlow installed from (source or binary): Binary
• TensorFlow version (use command below): 2.7.0 - 2.10.0
• Python version: Python 3.8
• Exact command to reproduce: See Colabs below

Describe the problem.

I ran the official Keras tutorial "Structured data classification from scratch" in TF 2.7, and saved the model as a SavedModel.

This SavedModel successfully loads in TF 2.7, but cannot be loaded TF 2.8, 2.9, or 2.10. Error message:

File ~/.../regressor.py:337, in load_model(filepath, library)
--> 337             model = tf.keras.models.load_model(extract_dir)

File ~/.../env/lib/python3.8/site-packages/keras/utils/traceback_utils.py:67, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     65 except Exception as e:  # pylint: disable=broad-except
     66   filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67   raise e.with_traceback(filtered_tb) from None
     68 finally:
     69   del filtered_tb

File ~/.../env/lib/python3.8/site-packages/tensorflow/python/framework/op_def_library.py:56, in _SatisfiesTypeConstraint(dtype, attr_def, param_name)
     54 allowed_values = ", ".join(dtypes.as_dtype(x).name for x in allowed_list)
     55 if dtype not in allowed_list:
---> 56   raise TypeError(
     57       f"Value passed to parameter '{param_name}' has DataType "
     58       f"{dtypes.as_dtype(dtype).name} not in list of allowed values: "
     59       f"{allowed_values}")

TypeError: Exception encountered when calling layer "string_lookup" (type StringLookup).

Value passed to parameter 'weights' has DataType string not in list of allowed values: int32, int64, float32, float64

Call arguments received:
  • inputs=tf.Tensor(shape=(None, 1), dtype=string)

Based my my tests, here's a compatibility table for saving/loading this model across TF versions:

	Load with TF 2.6	Load with TF 2.7	Load with TF 2.8	Load with TF 2.9	Load with TF 2.10
Saved with TF 2.6	✅	✅	❌	❌	❌
Saved with TF 2.7	❌	✅	❌	❌	❌
Saved with TF 2.8	❌	❌	✅	✅	✅
Saved with TF 2.9	❌	❌	✅	✅	✅
Saved with TF 2.10	❌	❌	✅	✅	✅

Standalone code to reproduce the issue.

Colab to train and save the Keras tutorial model: https://colab.research.google.com/drive/1c-UDWPW0OQjbgItijs_4UB4sm-dQ7MdS?usp=sharing

Colab to load the Keras tutorial SavedModel: https://colab.research.google.com/drive/1KM9nUBdsqGFW5zXUuU7b2ZtcKA1BiUXz?usp=sharing

[tilakrayal]

@gowthamkpr, I was able to reproduce the issue on tensorflow v2.8, v2.9, v2.10 and nightly. Kindly find the gist of it here. Model Gist, Gist1 where as on v2.7 the model was able to load. Thank you!

[gowthamkpr]

Hi Kenny, this is been fixed in the latest version of tf-nightly and it works. Keras team is coming up with a new format that ends with .keras extension. This works in both keras format .keras and saved model .tf format. Heres the gist. Try running it and it should work. Thanks!!

[kennysong]

Hi @gowthamkpr, thanks for the response!

Unfortunately, this does not resolve the issue. tf-nightly (2.12.0-dev20221114) is still unable to load the SavedModels from TF 2.7 and 2.6.

The problem is that the SavedModel is incompatible across TF versions. There is no problem loading a SavedModel with the same version of TF used to generate it.

[mattdangerw]

Thanks for filing! Will take a deeper look.

[tilakrayal]

@kennysong, Since the migration, there were many changes made, the error you are getting is due to one such change here https://github.com/keras-team/tf-keras/commit/cff6ac903e2b8a0dde2a469d949f0f0ce3b5f282.

To get rid of the error, you need to install tf-keras-nightly as well as tf-nightly.

Note that, tf-keras-nightly is legacy Keras code, to use the Keras 3 with multi-backend support, use keras-nightly and import Keras directly.

Also I tried to execute the Keras tutorial on tensorflow v2.14 and tried loading on v2.15 and it happened without any issues. Kindly find the gist of it here. Thank you!

[github-actions[bot]]

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[kennysong]

Hi @tilakrayal, I tried loading the tutorial TF 2.7 SavedModel in TF 2.14, 2.15, and tf-nightly + tf-keras-nightly, and none of them worked. (Here's the TF 2.7 SavedModel file which you can replicate from the notebook linked in the first comment.)

In TF 2.14 and 2.15, the error is the same as previously reported.

In TF nightly (Keras 3), SavedModels cannot be directly loaded as noted in the migration guide. This wrapper code seems to work:

m = keras.Sequential(
    [keras.layers.TFSMLayer('tutorial_model_tf26', call_endpoint="serving_default")]
)

But I wasn't able to easily get the loaded model to make a prediction due to lack of documentation. Pointers would be appreciated!

[github-actions[bot]]

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[kennysong]

This issue is not stale.