[kws_streaming] export stream_state_internal savedmodel stuck at keras.engine.functional.reconstruct_from_config

[IvyTang]

system: ubuntu 20.04 tensorflow: 2.9.1

Based on the experiments/ , I have success in train & export saved model of non_stream . Tf2.9.1 is the only version I tried can train the models with some little fixed , I can't find tf_nightly-2.3.0.dev20200515 in the experiments/ 。 But when i tried to export saved model of stream_state_internal. I found it stuck at reconstruct_from_config , the node_index_map is always empty ,so node_index_map return always None. It keeps add node into unprocessed_nodes. I print some variables . The unprocessed_nodes is

{<kws_streaming.layers.speech_features.SpeechFeatures object at 0x7ff1804fb730>:[[<tensorflow.python.keras.utils.tf_utils.ListWrapper object at 0x7ff18056d940>]], 
<kws_streaming.layers.lstm.LSTM object at 0x7ff1804fbbb0>: 
[[<tensorflow.python.keras.utils.tf_utils.ListWrapper object at 0x7ff18056dc40>]], 
<kws_streaming.layers.stream.Stream object at 0x7ff1804fbb20>:
 [[<tensorflow.python.keras.utils.tf_utils.ListWrapper object at 0x7ff18056dc10>]], 
<keras.layers.regularization.dropout.Dropout object at 0x7ff180229e50>:
 [[<tensorflow.python.keras.utils.tf_utils.ListWrapper object at 0x7ff18056d400>]], 
<keras.layers.core.dense.Dense object at 0x7ff18069bf40>: 
[[<tensorflow.python.keras.utils.tf_utils.ListWrapper object at 0x7ff18056d430>]]}

And my export script is:

python -m kws_streaming.train.model_train_eval --data_url '' \
--data_dir kws_streaming/data2 \
--train_dir  kws_streaming/lstm_peep/ \
--how_many_training_steps 20000,20000,20000,20000 \
--learning_rate 0.001,0.0005,0.0001,0.00002 \
--window_size_ms 40.0 \
--window_stride_ms 20.0 \
--mel_num_bins 40 \
--dct_num_features 20 \
--resample 0.15 \
--alsologtostderr \
--train 0 \
--lr_schedule 'exp' \
--use_spec_augment 1 \
--time_masks_number 2 \
--time_mask_max_size 10 \
--frequency_masks_number 2 \
--frequency_mask_max_size 5 \
lstm \
--lstm_units 500 \
--return_sequences 0 \
--use_peepholes 1 \
--num_proj 200 \
--dropout1 0.3 \
--units1 '' \
--act1 '' \
--stateful 0

How to solve it ? Thanks!

[rybakov]

@IvyTang even if you find version tf_nightly-2.3.0.dev20200515 you will have to use the corresponding version of kws_streaming at github. For example you can pick a version which is close to tf_nightly-2.3.0.dev20200515 at link and kws_streaming version of Oct 23, 2020, e.g. commit). kws_streaming is always at sync with the latest version of TF, so it is better to use TF nightly version. I would suggest to try another model e.g. 'gru' and let me know if it works (there is no need to train it long: train for 1000 iterations and check that it can be converted).

[caveman77]

Hello @rybakov

it seems that tf_nightly-2.3.0.dev20200515 cannot be downloaded anymore. I used the last google research package release with TF 2.14 and I got the same issue as mentioned here : https://github.com/google-research/google-research/issues/1753.

Could you please either tell us where to find the tf_nightly release or which TF release has been correctly been tested successfully with the last kws_streaming ?

thanks a lot !

[Jacob-Bishop]

Looks like someone filed an issue with tensorflow about this infinite loop in clone_model here, and the approved solution is to migrate code to use keras3 instead of tensorflow.keras.