PaddleHub
PaddleHub copied to clipboard
Published
20 hours ago •
PaddlePaddle
PaddlePaddle
Cannot access deeplabv3p_xception65_humanseg
Environment:
Python 3.9.13 paddle-bfloat 0.1.7 paddle2onnx 1.0.1 paddlefsl 1.1.0 paddlehub 2.3.0 paddlenlp 2.4.1 paddlepaddle 2.3.2 paddleseg 2.6.0
Windows 10
Problem: I tried to convert deeplabv3p_xception65_humanseg to ONNX. However, two different ways to load the model failed.
- According to https://github.com/PaddlePaddle/PaddleHub/blob/develop/modules/image/semantic_segmentation/deeplabv3p_xception65_humanseg/README_en.md use
(.venv) D:\Local\devel\Python\PaddleSeg>hub install deeplabv3p_xception65_humanseg
Traceback (most recent call last):
File "D:\Local\Python\Python3.9.13\lib\runpy.py", line 197, in _run_module_as_
main
return _run_code(code, main_globals, None,
File "D:\Local\Python\Python3.9.13\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "D:\Local\devel\Python\PaddleSeg\.venv\Scripts\hub.exe\__main__.py", line
7, in <module>
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\comman
ds\utils.py", line 78, in execute
status = 0 if com['_entry']().execute(sys.argv[idx:]) else 1
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\comman
ds\install.py", line 47, in execute
manager.install(directory=_arg)
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module
\manager.py", line 192, in install
return self._install_from_directory(directory)
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module
\manager.py", line 342, in _install_from_directory
module_info = HubModule.load_module_info(directory)
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module
\module.py", line 457, in load_module_info
with codecs.open(module_file, 'r', encoding='utf8') as file:
File "D:\Local\Python\Python3.9.13\lib\codecs.py", line 905, in open
file = builtins.open(filename, mode, buffering)
FileNotFoundError: [Errno 2] No such file or directory: 'deeplabv3p_xception65_h
umanseg\\module.py'
- Tried to load in a python script using:
model_name = "deeplabv3p_xception65_humanseg"
save_path = './' + model_name
model = hub.Module(name=model_name)
paddle.onnx.export(model, save_path,
# input_spec=[input_spec],
opset_version=11)
results in
Download https://bj.bcebos.com/paddlehub/paddlehub_dev/deeplabv3p_xception65_humanseg_1.1.2.tar.gz
[##################################################] 100.00%
Decompress C:\Users\ukoehler\.paddlehub\tmp\tmp9n4mycm6\deeplabv3p_xception65_humanseg_1.1.2.tar.gz
<traceback object at 0x0000021BD22A7080>
Traceback (most recent call last):
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\utils\xarfile.py", line 228, in unarchive_with_progress
total_size += file.getxarinfo(filename).size
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\utils\xarfile.py", line 153, in getxarinfo
return XarInfo(self._archive_fp.getmember(name), self.arctype)
File "D:\Local\Python\Python3.9.13\lib\tarfile.py", line 1799, in getmember
raise KeyError("filename %r not found" % name)
KeyError: "filename 'deeplabv3p_xception65_humanseg/deeplabv3p_xception65_humanseg_model/decoder/separable_conv2/pointwise/' not found"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\utils\utils.py", line 185, in generate_tempdir
yield _dir
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module\manager.py", line 258, in _install_from_url
return self._install_from_archive(file)
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module\manager.py", line 374, in _install_from_archive
for path, ds, ts in xarfile.unarchive_with_progress(archive, _tdir):
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\utils\xarfile.py", line 233, in unarchive_with_progress
yield filename, extract_size, total_size
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\utils\xarfile.py", line 109, in __exit__
raise exit_exception(exit_value)
KeyError: KeyError("filename 'deeplabv3p_xception65_humanseg/deeplabv3p_xception65_humanseg_model/decoder/separable_conv2/pointwise/' not found")
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Local\Python\Python3.9.13\lib\shutil.py", line 627, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 32] Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird: 'C:\\Users\\ukoehler\\.paddlehub\\tmp\\tmp9n4mycm6\\deeplabv3p_xception65_humanseg_1.1.2.tar.gz'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Local\Python\Python3.9.13\lib\tempfile.py", line 805, in onerror
_os.unlink(path)
PermissionError: [WinError 32] Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird: 'C:\\Users\\ukoehler\\.paddlehub\\tmp\\tmp9n4mycm6\\deeplabv3p_xception65_humanseg_1.1.2.tar.gz'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "d:\Local\devel\Python\PaddleSeg\deeplabv3p_xception65_humanseg\convert_to_onnx.py", line 47, in <module>
model = hub.Module(name=model_name)
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module\module.py", line 390, in __new__
module = cls.init_with_name(name=name,
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module\module.py", line 488, in init_with_name
user_module_cls = manager.install(name=name,
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module\manager.py", line 190, in install
return self._install_from_name(name, version, ignore_env_mismatch)
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module\manager.py", line 265, in _install_from_name
return self._install_from_url(item['url'])
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module\manager.py", line 258, in _install_from_url
return self._install_from_archive(file)
File "D:\Local\Python\Python3.9.13\lib\contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\utils\utils.py", line 185, in generate_tempdir
yield _dir
File "D:\Local\Python\Python3.9.13\lib\tempfile.py", line 830, in __exit__
self.cleanup()
File "D:\Local\Python\Python3.9.13\lib\tempfile.py", line 834, in cleanup
self._rmtree(self.name)
File "D:\Local\Python\Python3.9.13\lib\tempfile.py", line 816, in _rmtree
_shutil.rmtree(name, onerror=onerror)
File "D:\Local\Python\Python3.9.13\lib\shutil.py", line 759, in rmtree
return _rmtree_unsafe(path, onerror)
File "D:\Local\Python\Python3.9.13\lib\shutil.py", line 629, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "D:\Local\Python\Python3.9.13\lib\tempfile.py", line 808, in onerror
cls._rmtree(path)
File "D:\Local\Python\Python3.9.13\lib\tempfile.py", line 816, in _rmtree
_shutil.rmtree(name, onerror=onerror)
File "D:\Local\Python\Python3.9.13\lib\shutil.py", line 759, in rmtree
return _rmtree_unsafe(path, onerror)
File "D:\Local\Python\Python3.9.13\lib\shutil.py", line 610, in _rmtree_unsafe
onerror(os.scandir, path, sys.exc_info())
File "D:\Local\Python\Python3.9.13\lib\shutil.py", line 607, in _rmtree_unsafe
with os.scandir(path) as scandir_it:
NotADirectoryError: [WinError 267] Der Verzeichnisname ist ungültig: 'C:\\Users\\ukoehler\\.paddlehub\\tmp\\tmp9n4mycm6\\deeplabv3p_xception65_humanseg_1.1.2.tar.gz'
The first error is strange, since the folder 'deeplabv3p_xception65_humanseg/deeplabv3p_xception65_humanseg_model/decoder/separable_conv2/pointwise/' is indeed available in the *.tar.gz file. The secondary error messages reports the *.tar.gz file being locked by another process. This means that the temporary file is not deleted.
Can you point me in the right direction to fix the problem, please?
Just in case it helps: installing humanseg_server works fine
(.venv) D:\Local\devel\Python\PaddleSeg>hub install humanseg_server
Download https://bj.bcebos.com/paddlehub/paddlehub_dev/humanseg_server.tar.gz
[##################################################] 100.00%
Decompress C:\Users\ukoehler\.paddlehub\tmp\tmpj6g_ii9x\humanseg_server.tar.gz
[##################################################] 100.00%
[2022-10-20 14:38:40,847] [ INFO] - Successfully installed humanseg_server-1.
2.1
Environment:
Python 3.9.13 paddle-bfloat 0.1.7 paddle2onnx 1.0.1 paddlefsl 1.1.0 paddlehub 2.3.0 paddlenlp 2.4.1 paddlepaddle 2.3.2 paddleseg 2.6.0
Windows 10
Problem: I tried to convert deeplabv3p_xception65_humanseg to ONNX. However, two different ways to load the model failed.
- According to https://github.com/PaddlePaddle/PaddleHub/blob/develop/modules/image/semantic_segmentation/deeplabv3p_xception65_humanseg/README_en.md use
(.venv) D:\Local\devel\Python\PaddleSeg>hub install deeplabv3p_xception65_humanseg Traceback (most recent call last): File "D:\Local\Python\Python3.9.13\lib\runpy.py", line 197, in _run_module_as_ main return _run_code(code, main_globals, None, File "D:\Local\Python\Python3.9.13\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "D:\Local\devel\Python\PaddleSeg\.venv\Scripts\hub.exe\__main__.py", line 7, in <module> File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\comman ds\utils.py", line 78, in execute status = 0 if com['_entry']().execute(sys.argv[idx:]) else 1 File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\comman ds\install.py", line 47, in execute manager.install(directory=_arg) File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module \manager.py", line 192, in install return self._install_from_directory(directory) File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module \manager.py", line 342, in _install_from_directory module_info = HubModule.load_module_info(directory) File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module \module.py", line 457, in load_module_info with codecs.open(module_file, 'r', encoding='utf8') as file: File "D:\Local\Python\Python3.9.13\lib\codecs.py", line 905, in open file = builtins.open(filename, mode, buffering) FileNotFoundError: [Errno 2] No such file or directory: 'deeplabv3p_xception65_h umanseg\\module.py'
- Tried to load in a python script using:
model_name = "deeplabv3p_xception65_humanseg" save_path = './' + model_name model = hub.Module(name=model_name) paddle.onnx.export(model, save_path, # input_spec=[input_spec], opset_version=11)results in
Download https://bj.bcebos.com/paddlehub/paddlehub_dev/deeplabv3p_xception65_humanseg_1.1.2.tar.gz [##################################################] 100.00% Decompress C:\Users\ukoehler\.paddlehub\tmp\tmp9n4mycm6\deeplabv3p_xception65_humanseg_1.1.2.tar.gz <traceback object at 0x0000021BD22A7080> Traceback (most recent call last): File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\utils\xarfile.py", line 228, in unarchive_with_progress total_size += file.getxarinfo(filename).size File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\utils\xarfile.py", line 153, in getxarinfo return XarInfo(self._archive_fp.getmember(name), self.arctype) File "D:\Local\Python\Python3.9.13\lib\tarfile.py", line 1799, in getmember raise KeyError("filename %r not found" % name) KeyError: "filename 'deeplabv3p_xception65_humanseg/deeplabv3p_xception65_humanseg_model/decoder/separable_conv2/pointwise/' not found" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\utils\utils.py", line 185, in generate_tempdir yield _dir File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module\manager.py", line 258, in _install_from_url return self._install_from_archive(file) File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module\manager.py", line 374, in _install_from_archive for path, ds, ts in xarfile.unarchive_with_progress(archive, _tdir): File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\utils\xarfile.py", line 233, in unarchive_with_progress yield filename, extract_size, total_size File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\utils\xarfile.py", line 109, in __exit__ raise exit_exception(exit_value) KeyError: KeyError("filename 'deeplabv3p_xception65_humanseg/deeplabv3p_xception65_humanseg_model/decoder/separable_conv2/pointwise/' not found") During handling of the above exception, another exception occurred: Traceback (most recent call last): File "D:\Local\Python\Python3.9.13\lib\shutil.py", line 627, in _rmtree_unsafe os.unlink(fullname) PermissionError: [WinError 32] Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird: 'C:\\Users\\ukoehler\\.paddlehub\\tmp\\tmp9n4mycm6\\deeplabv3p_xception65_humanseg_1.1.2.tar.gz' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "D:\Local\Python\Python3.9.13\lib\tempfile.py", line 805, in onerror _os.unlink(path) PermissionError: [WinError 32] Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird: 'C:\\Users\\ukoehler\\.paddlehub\\tmp\\tmp9n4mycm6\\deeplabv3p_xception65_humanseg_1.1.2.tar.gz' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "d:\Local\devel\Python\PaddleSeg\deeplabv3p_xception65_humanseg\convert_to_onnx.py", line 47, in <module> model = hub.Module(name=model_name) File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module\module.py", line 390, in __new__ module = cls.init_with_name(name=name, File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module\module.py", line 488, in init_with_name user_module_cls = manager.install(name=name, File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module\manager.py", line 190, in install return self._install_from_name(name, version, ignore_env_mismatch) File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module\manager.py", line 265, in _install_from_name return self._install_from_url(item['url']) File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\module\manager.py", line 258, in _install_from_url return self._install_from_archive(file) File "D:\Local\Python\Python3.9.13\lib\contextlib.py", line 137, in __exit__ self.gen.throw(typ, value, traceback) File "D:\Local\devel\Python\PaddleSeg\.venv\lib\site-packages\paddlehub\utils\utils.py", line 185, in generate_tempdir yield _dir File "D:\Local\Python\Python3.9.13\lib\tempfile.py", line 830, in __exit__ self.cleanup() File "D:\Local\Python\Python3.9.13\lib\tempfile.py", line 834, in cleanup self._rmtree(self.name) File "D:\Local\Python\Python3.9.13\lib\tempfile.py", line 816, in _rmtree _shutil.rmtree(name, onerror=onerror) File "D:\Local\Python\Python3.9.13\lib\shutil.py", line 759, in rmtree return _rmtree_unsafe(path, onerror) File "D:\Local\Python\Python3.9.13\lib\shutil.py", line 629, in _rmtree_unsafe onerror(os.unlink, fullname, sys.exc_info()) File "D:\Local\Python\Python3.9.13\lib\tempfile.py", line 808, in onerror cls._rmtree(path) File "D:\Local\Python\Python3.9.13\lib\tempfile.py", line 816, in _rmtree _shutil.rmtree(name, onerror=onerror) File "D:\Local\Python\Python3.9.13\lib\shutil.py", line 759, in rmtree return _rmtree_unsafe(path, onerror) File "D:\Local\Python\Python3.9.13\lib\shutil.py", line 610, in _rmtree_unsafe onerror(os.scandir, path, sys.exc_info()) File "D:\Local\Python\Python3.9.13\lib\shutil.py", line 607, in _rmtree_unsafe with os.scandir(path) as scandir_it: NotADirectoryError: [WinError 267] Der Verzeichnisname ist ungültig: 'C:\\Users\\ukoehler\\.paddlehub\\tmp\\tmp9n4mycm6\\deeplabv3p_xception65_humanseg_1.1.2.tar.gz'The first error is strange, since the folder 'deeplabv3p_xception65_humanseg/deeplabv3p_xception65_humanseg_model/decoder/separable_conv2/pointwise/' is indeed available in the *.tar.gz file. The secondary error messages reports the *.tar.gz file being locked by another process. This means that the temporary file is not deleted.
Can you point me in the right direction to fix the problem, please?
This module may currently have problems running on windows systems.
model_name = "deeplabv3p_xception65_humanseg" save_path = './' + model_name model = hub.Module(name=model_name) paddle.onnx.export(model, save_path, # input_spec=[input_spec], opset_version=11)
Most of the current PaddleHub modules cannot be exported to ONNX models using this method either.
deeplabv3p_xception65_humanseg module uses an old version of the format, and currently the paddle2onnx project cannot convert it.
[ERROR] Cannot found attribute padding_algorithm in op: conv2d
You may want to try exporting the model using the PaddleSeg project and converting it to ONNX format.
I tried the export, but the script doesn't do anything when following the instructions. See https://github.com/PaddlePaddle/PaddleSeg/issues/2643
@jm12138 I got the PaddleSeg ONNX export to work. However, where would I get the model.pdparams and the corresponding *.yml file?
A bit of a shame, since the examples look really impressive.
After a paddlehub up date the install works now:
(.venv) D:\Local\devel\Python\PaddleSeg_2.7.0>hub install deeplabv3p_xception65_
humanseg
Download https://bj.bcebos.com/paddlehub/paddlehub_dev/deeplabv3p_xception65_hum
anseg_1_2_0.zip
[##################################################] 100.00%
Decompress C:\Users\ukoehler\.paddlehub\tmp\tmptznv48kc\deeplabv3p_xception65_hu
manseg_1_2_0.zip
[##################################################] 100.00%
[2023-01-10 12:09:39,796] [ INFO] - Successfully installed deeplabv3p_xceptio
n65_humanseg-1.2.0
The update got me a bit further. The ONNX export does now fail, however: https://github.com/PaddlePaddle/Paddle2ONNX/issues/1008