serve icon indicating copy to clipboard operation
serve copied to clipboard

Keep root dir for extra files when generating .mar file in model-archiver

Open csJoax opened this issue 3 years ago • 0 comments

Description

Please read our CONTRIBUTING.md prior to creating your first pull request.

Please include a summary of the feature or issue being fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #1689

Type of change

  • [x] New feature (non-breaking change which adds functionality)
  • [ ] Documentation updated. see README in examples/mar_extra_root/.

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced. Please also list any relevant details for your test configuration.

  • [x] Test A
$ torch-model-archiver \
    --model-name mnist \
    --version 1.0 \
    --model-file      examples/image_classifier/mnist/mnist.py \
    --serialized-file examples/image_classifier/mnist/mnist_cnn.pt \
    --handler        'examples/mar_extra_root/mnist_handler_extra.py' \
   --extra-files     'examples/mar_extra_root/my_lib1,examples/mar_extra_root/my_lib2' 
ERROR - Maybe running with `--keep-extra-root` or rename the files
Traceback (most recent call last):
  File "/opt/miniconda3/envs/torch/bin/torch-model-archiver", line 33, in <module>
    sys.exit(load_entry_point('torch-model-archiver', 'console_scripts', 'torch-model-archiver')())
  File "/home/joax/GitSpace/pytorch_serve/model-archiver/model_archiver/model_packaging.py", line 57, in generate_model_archive
    package_model(args, manifest=manifest)
  File "/home/joax/GitSpace/pytorch_serve/model-archiver/model_archiver/model_packaging.py", line 36, in package_model
    model_path = ModelExportUtils.copy_artifacts(model_name, keep_extra_root=args.keep_extra_root,
  File "/home/joax/GitSpace/pytorch_serve/model-archiver/model_archiver/model_packaging_utils.py", line 180, in copy_artifacts
    ModelExportUtils.__copy_dir(file, model_path, keep_extra_root)
  File "/home/joax/GitSpace/pytorch_serve/model-archiver/model_archiver/model_packaging_utils.py", line 146, in __copy_dir
    shutil.copytree(src, dst, False, None)
  File "/opt/miniconda3/envs/torch/lib/python3.9/shutil.py", line 565, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/opt/miniconda3/envs/torch/lib/python3.9/shutil.py", line 466, in _copytree
    os.makedirs(dst, exist_ok=dirs_exist_ok)
  File "/opt/miniconda3/envs/torch/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/tmp/mnist/my_module'
  • [x] Test B
$ torch-model-archiver \
    --model-name mnist \
    --version 1.0 \
    --model-file      examples/image_classifier/mnist/mnist.py \
    --serialized-file examples/image_classifier/mnist/mnist_cnn.pt \
    --handler         'examples/mar_extra_root/mnist_handler_extra.py' \
    --extra-files     'examples/mar_extra_root/my_lib1,examples/mar_extra_root/my_lib2' \
    --keep-extra-root
$ unzip -l mnist.mar
Archive:  mnist.mar
  Length      Date    Time    Name
---------  ---------- -----   ----
     1376  2022-06-19 02:06   mnist_handler_extra.py
  4800893  2022-06-19 02:06   mnist_cnn.pt
      757  2022-06-19 02:06   mnist.py
       58  2022-06-19 01:32   my_lib2/my_file.py
       59  2022-06-19 01:32   my_lib2/my_module/my_file1.py
       58  2022-06-19 01:32   my_lib1/my_file.py
       59  2022-06-19 01:32   my_lib1/my_module/my_file1.py
      271  2022-06-19 02:06   MAR-INF/MANIFEST.json
---------                     -------
  4803531                     8 files

Checklist:

  • [x] Did you have fun?
  • [x] Have you added tests that prove your fix is effective or that this feature works? Yes, see examples/mar_extra_root/
  • [x] Has code been commented, particularly in hard-to-understand areas?
  • [x] Have you made corresponding changes to the documentation? Yes, see README in examples/mar_extra_root/

csJoax avatar Jun 19 '22 04:06 csJoax