tensorflowjs_converter on .keras file messes up weights

[smidm]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js): yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Fedora 39
• TensorFlow.js installed from (npm or script link): pypi
• TensorFlow.js version (use command below): 4.18.0
• keras 2.15.1
• tensorflow 2.15.1
• tf_keras 2.15.1

Describe the current behavior

.keras saved model after being converted to tfjs_graph_model format using tensorflowjs_converter has different weight values when loaded using tensorflow.js.

create a dummy model:

import keras
from tensorflow.keras import layers, models

dummy_model = models.Sequential([
        layers.Input(shape=(None, 3)),
        layers.GlobalMaxPooling1D(),
        layers.Dense(4, activation='softmax')
    ])
dummy_model.save("dummy_model.keras")
print(dummy_model.get_weights())

output:

[
array([[-0.72, -0.83, -0.59,  0.12],
       [ 0.6 ,  0.59, -0.73, -0.18],
       [-0.49, -0.49, -0.09, -0.43]], dtype=float32), 
array([0., 0., 0., 0.], dtype=float32)
]

convert to tfjs_graph_model format:

tensorflowjs_converter --input_format=keras_saved_model dummy_model.keras model/

output:

2024-05-01 17:18:13.812215: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2024-05-01 17:18:13.879791: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-05-01 17:18:13.879828: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-05-01 17:18:13.890816: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-05-01 17:18:13.915374: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2024-05-01 17:18:13.915578: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-05-01 17:18:15.270355: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
WARNING:root:TensorFlow Decision Forests 1.8.1 is compatible with the following TensorFlow Versions: ['2.15.0']. However, TensorFlow 2.15.1 was detected. This can cause issues with the TF API and symbols in the custom C++ ops. See the TF and TF-DF compatibility table at https://github.com/tensorflow/decision-forests/blob/main/documentation/known_issues.md#compatibility-table.
/home/matej/local/conda/envs/pose_service/lib/python3.9/site-packages/tf_keras/src/engine/training.py:3098: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native TF-Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(
2024-05-01 17:18:18.201563: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:388] MLIR V1 optimization pass is not enabled
2024-05-01 17:18:18.215232: W tensorflow/c/c_api.cc:305] Operation '{name:'dense_1/bias/Assign' id:22 op device:{requested: '', assigned: ''} def:{{{node dense_1/bias/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](dense_1/bias, dense_1/bias/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.

load model in JS:

async function loadModel() {
    try {
        // Load the model from the local server
        const model = await tf.loadLayersModel('model/model.json');
        console.log('Model loaded successfully');

        for (let i = 0; i < model.getWeights().length; i++) {
            console.log(model.getWeights()[i].dataSync());
        }

    } catch (error) {
        console.error('Failed to load model or make a prediction:', error);
    }    
}

loadModel();

output:

Model loaded successfully

Float32Array(12) [ -0.5237351655960083, -0.5906776189804077, -0.4140402674674988, 0.3560253381729126, 0.080802321434021, -0.19059759378433228, 0.22761249542236328, 0.843163013458252, -0.35245829820632935, -0.1886269450187683, … ]
​
0: -0.5237351655960083​
1: -0.5906776189804077​
2: -0.4140402674674988​
3: 0.3560253381729126​
4: 0.080802321434021​
5: -0.19059759378433228​
6: 0.22761249542236328​
7: 0.843163013458252​
8: -0.35245829820632935​
9: -0.1886269450187683​
10: -0.8403407335281372​
11: -0.13888061046600342
​
Float32Array(4) [ 0, 0, 0, 0 ]

Describe the expected behavior

The weights of the python model and the model loaded in JS should be the same or should tensorflowjs_converter end with an error if .keras model format is not supported https://github.com/tensorflow/tfjs/issues/8229#issuecomment-2029760312 .

Standalone code to reproduce the issue

.keras model, tfjs_graph_model and sample JS app to load the model: tensorflowjs_converter_issue.zip

[gaikwadrahul8]

Hi, @smidm

I apologize for the delayed response and as far I know the TensorFlow.js converter currently does not support converting Keras models saved in the new format .keras . it's still relatively new and hasn't been fully integrated with the TensorFlow.js converter yet so I completely agree with you in my view, tensorflowjs_converter should provide a user-friendly error message explicitly stating its lack of support for the .keras format. An informative message such as " .keras file format is not currently supported by tensorflowjs_converter. Please refer to the documentation for supported formats" would enhance the user experience by offering clear guidance.

Meanwhile, as an alternative approach, could you explore exporting the model directly to TensorFlow.js Layers format using the Python API? This would allow us to compare the weights of the original Python model and the converted model. I'll discuss this issue within our internal meeting and will update you soon. thank you for bringing this issue to our attention, I really appreciate your efforts and valuable time.

# Python

import tensorflowjs as tfjs

def train(...):
    model = keras.models.Sequential()   # for example
    ...
    model.compile(...)
    model.fit(...)
    tfjs.converters.save_keras_model(model, tfjs_target_dir)

Thank you for your cooperation and patience.

[smidm]

Thanks for the alternative approach. I have already figured out, before submitting the issue, but it took some time. IMO the tensorflowjs_converter should check if inputs are valid and throw an error if not. It'll save a lot of pain. I had similar problem in https://github.com/tensorflow/tfjs/issues/8265.

[gaikwadrahul8]

Hi, @smidm

Thank you for your suggestion regarding to add a link https://github.com/tensorflow/tfjs/tree/master/tfjs-converter to tensorflowjs_converter --help . I appreciate you bringing this to our attention.

I will be sure to share your feedback during our next internal meeting. The team will discuss it and take appropriate action if necessary.

If your issue has been resolved, please feel free to close this issue.

Thank you for your cooperation and patience.

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No