bojone
model.save() 报错TypeError: can't pickle _thread.RLock objects
提问时请尽可能提供如下信息：
基本信息
- 你使用的操作系统: linux
- 你使用的Python版本: 3.7
- 你使用的Tensorflow版本: 2.1.0
- 你使用的Keras版本: 2.3.1
- 你使用的bert4keras版本: 0.10.0
- 你使用纯keras还是tf.keras: tf.keras
- 你加载的预训练模型: t5-pegasus
核心代码
t5 = build_transformer_model(
config_path=config_path,
checkpoint_path=checkpoint_path,
model='t5.1.1',
return_keras_model=False,
name='T5',
)
encoder = t5.encoder
decoder = t5.decoder
model = t5.model
# model.summary()
output = CrossEntropy(1)([model.inputs[1], model.outputs[0]])
model = Model(model.inputs, output)
# model.compile(optimizer=Adam(2e-4))
model.load_weights(f'{args.load_model_path}/best_model.weights')
model.save('path_to_save/model.h5')
输出信息
Traceback (most recent call last):
File "convert_ckpt2pb.py", line 152, in <module>
model.save('path_to_save/model.h5')
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 1008, in save
signatures, options)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/save.py", line 112, in save_model
model, filepath, overwrite, include_optimizer)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 99, in save_model_to_hdf5
model_metadata = saving_utils.model_metadata(model, include_optimizer)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saving_utils.py", line 169, in model_metadata
model_config['config'] = model.get_config()
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 918, in get_config
return copy.deepcopy(get_network_config(self))
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 1993, in get_network_config
layer_config = serialize_layer_fn(layer)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/generic_utils.py", line 198, in serialize_keras_object
config = instance.get_config()
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 918, in get_config
return copy.deepcopy(get_network_config(self))
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 241, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 216, in _deepcopy_list
append(deepcopy(a, memo))
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 241, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 241, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 221, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 221, in <listcomp>
y = [deepcopy(a, memo) for a in x]
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 221, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 221, in <listcomp>
y = [deepcopy(a, memo) for a in x]
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 281, in _reconstruct
state = deepcopy(state, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 241, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 281, in _reconstruct
state = deepcopy(state, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 241, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 281, in _reconstruct
state = deepcopy(state, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 241, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 281, in _reconstruct
state = deepcopy(state, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 241, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/export/zouyanyan/anaconda3/envs/py3.7-tf2.2/lib/python3.7/copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: can't pickle _thread.RLock objects
自我尝试
尝试过
1> tf1.15.0的版本
2> 是否启用os.environ['TF_KERAS'] = '1'，
3> tf.keras.models.save_model(model,'path_to_save/model.h5', save_format='tf')
都会报以上error
谢谢苏神
单独保存encoder或者decoder呢
感谢苏神回复，按照苏神的建议，代码小改，如下：
` os.environ['TF_KERAS'] = '1'
encoder = t5.encoder decoder = t5.decoder model = t5.model
model = Model(encoder.inputs, encoder.outputs) tf.keras.models.save_model(model,'path_to_save/model.h5', save_format='tf') ` 程序可以正常执行，得到model.h5文件，不过保存的model还是keras的h5格式，并不是tf的savedmodel 格式。
如果将encoder，换成decoder，还是报上面的错误
TypeError: can't pickle _thread.RLock objects
请苏神指点迷津，谢谢
单独保存encoder或者decoder呢
感谢苏神回复，按照苏神的建议，代码小改，如下：
` os.environ['TF_KERAS'] = '1'
encoder = t5.encoder decoder = t5.decoder model = t5.model
model = Model(encoder.inputs, encoder.outputs) tf.keras.models.save_model(model,'path_to_save/model.h5', save_format='tf') ` 程序可以正常执行，得到model.h5文件，不过保存的model还是keras的h5格式，并不是tf的savedmodel 格式。
如果将encoder，换成decoder，还是报上面的错误
TypeError: can't pickle _thread.RLock objects请苏神指点迷津，谢谢
只提供路径就可以得到savedmodel 格式， tf.keras.models.save_model(model,'path_to_save/model', save_format='tf')
没。。。
Esther @.***> 于2022年2月17日周四 11:21写道：
同学你解决了吗！ 遇到了同样的问题
— Reply to this email directly, view it on GitHub https://github.com/bojone/bert4keras/issues/346#issuecomment-1042533781, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADS456KMT7DZGKUYIWFATJDU3RSVJANCNFSM45O5525Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID: @.***>
没。。。 Esther @.> 于2022年2月17日周四 11:21写道： … 同学你解决了吗！ 遇到了同样的问题 — Reply to this email directly, view it on GitHub <#346 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADS456KMT7DZGKUYIWFATJDU3RSVJANCNFSM45O5525Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you authored the thread.Message ID: @.>
你好，我也遇到了同样的问题，环境也基本一样，现在有解决吗？这个模型部署问题很大呀
没。。。
Esther @.***> 于2022年2月17日周四 11:21写道：
同学你解决了吗！ 遇到了同样的问题
— Reply to this email directly, view it on GitHub https://github.com/bojone/bert4keras/issues/346#issuecomment-1042533781, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADS456KMT7DZGKUYIWFATJDU3RSVJANCNFSM45O5525Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID: @.***>