DeepSpeed icon indicating copy to clipboard operation
DeepSpeed copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

[BUG] pip install .

Open gabrielhuang opened this issue 2 years ago • 6 comments

Describe the bug pip install . returns with error

To Reproduce Steps to reproduce the behavior:

  1. Clone the repo, checkout commit 4575b2b792f6424d68677f9b896b55be9241aeac
  2. pip install .

Expected behavior Install master branch of deepspeed

ds_report output Can't

Screenshots image

System info (please complete the following information):

  • OS: Linux 691c0d12-8068-4c39-98d6-33c8b9144c31 5.4.0-99-generic #112-Ubuntu SMP Thu Feb 3 13:50:55 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
  • 1 GPU, V100-SXM2, CUDA 11.4, pytorch torch-1.10.2-cp39-cp39-manylinux1_x86_64.whl
  • Python version 3.9.7
  • Any other relevant info about your setup

Launcher context Terminal

gabrielhuang avatar Apr 22 '22 04:04 gabrielhuang

What version of pip are you using? (pip --version)

deepspeed/ops/csrc is a symbolic link to csrc in the top level of the repo, so not sure why pip would error out here.

Could you also try creating that symbolic link yourself and seeing if the install works? From the top level of the DeepSpeed repo, do ln -s $(pwd)/csrc $(pwd)/deepspeed/ops/csrc

mrwyattii avatar Apr 25 '22 16:04 mrwyattii

I'm using pip 20.3.1 from /mnt/home/miniconda3/lib/python3.9/site-packages/pip (python 3.9)

Oh i just got that warning maybe it's related: DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default. pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.

gabrielhuang avatar Apr 25 '22 16:04 gabrielhuang

I updated pip to latest 22.0.4 release, pulled from master and reran pip install ., and I created the symlink, I still get this error

Building wheels for collected packages: deepspeed
  Building wheel for deepspeed (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [27 lines of output]
      DS_BUILD_OPS=0
       [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
       [WARNING]  async_io: please install the libaio-dev package with apt
       [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
      Install Ops={'cpu_adam': False, 'cpu_adagrad': False, 'fused_adam': False, 'fused_lamb': False, 'sparse_attn': False, 'transformer': False, 'stochastic_transformer': False, 'async_io': False, 'utils': False, 'quantizer': False, 'transformer_inference': False}
      version=0.6.2+4575b2b7, git_hash=4575b2b7, git_branch=master
      install_requires=['hjson', 'ninja', 'numpy', 'packaging', 'psutil', 'py-cpuinfo', 'torch', 'tqdm', 'triton==1.0.0']
      compatible_ops={'cpu_adam': True, 'cpu_adagrad': True, 'fused_adam': True, 'fused_lamb': True, 'sparse_attn': True, 'transformer': True, 'stochastic_transformer': True, 'async_io': False, 'utils': True, 'quantizer': True, 'transformer_inference': True}
      ext_modules=[]
      running bdist_wheel
      running build
      running build_py
      copying deepspeed/git_version_info_installed.py -> build/lib/deepspeed
      running egg_info
      writing deepspeed.egg-info/PKG-INFO
      writing dependency_links to deepspeed.egg-info/dependency_links.txt
      writing entry points to deepspeed.egg-info/entry_points.txt
      writing requirements to deepspeed.egg-info/requires.txt
      writing top-level names to deepspeed.egg-info/top_level.txt
      reading manifest template 'MANIFEST.in'
      warning: no files found matching '*.hip' under directory 'deepspeed'
      warning: no files found matching '*.cc' under directory 'deepspeed'
      warning: no files found matching '*.tr' under directory 'csrc'
      warning: no files found matching '*.cc' under directory 'csrc'
      adding license file 'LICENSE'
      writing manifest file 'deepspeed.egg-info/SOURCES.txt'
      error: can't copy 'deepspeed/ops/csrc': doesn't exist or not a regular file
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for deepspeed
  Running setup.py clean for deepspeed
Failed to build deepspeed
Installing collected packages: deepspeed
  Attempting uninstall: deepspeed
    Found existing installation: deepspeed 0.5.10
    Uninstalling deepspeed-0.5.10:
      Successfully uninstalled deepspeed-0.5.10
  Running setup.py install for deepspeed ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for deepspeed did not run successfully.
  │ exit code: 1
  ╰─> [225 lines of output]
      DS_BUILD_OPS=0
       [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
       [WARNING]  async_io: please install the libaio-dev package with apt
       [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
      Install Ops={'cpu_adam': False, 'cpu_adagrad': False, 'fused_adam': False, 'fused_lamb': False, 'sparse_attn': False, 'transformer': False, 'stochastic_transformer': False, 'async_io': False, 'utils': False, 'quantizer': False, 'transformer_inference': False}
      version=0.6.2+4575b2b7, git_hash=4575b2b7, git_branch=master
      install_requires=['hjson', 'ninja', 'numpy', 'packaging', 'psutil', 'py-cpuinfo', 'torch', 'tqdm', 'triton==1.0.0']
      compatible_ops={'cpu_adam': True, 'cpu_adagrad': True, 'fused_adam': True, 'fused_lamb': True, 'sparse_attn': True, 'transformer': True, 'stochastic_transformer': True, 'async_io': False, 'utils': True, 'quantizer': True, 'transformer_inference': True}
      ext_modules=[]
      running install
      running build
      running build_py
      creating build
      creating build/lib
      creating build/lib/deepspeed
      copying deepspeed/__init__.py -> build/lib/deepspeed
      copying deepspeed/constants.py -> build/lib/deepspeed
      copying deepspeed/env_report.py -> build/lib/deepspeed
      copying deepspeed/git_version_info.py -> build/lib/deepspeed
      copying deepspeed/git_version_info_installed.py -> build/lib/deepspeed
      creating build/lib/deepspeed/autotuning
      copying deepspeed/autotuning/__init__.py -> build/lib/deepspeed/autotuning
      copying deepspeed/autotuning/autotuner.py -> build/lib/deepspeed/autotuning
      copying deepspeed/autotuning/config.py -> build/lib/deepspeed/autotuning
      copying deepspeed/autotuning/constants.py -> build/lib/deepspeed/autotuning
      copying deepspeed/autotuning/scheduler.py -> build/lib/deepspeed/autotuning
      copying deepspeed/autotuning/utils.py -> build/lib/deepspeed/autotuning
      creating build/lib/deepspeed/checkpoint
      copying deepspeed/checkpoint/__init__.py -> build/lib/deepspeed/checkpoint
      copying deepspeed/checkpoint/constants.py -> build/lib/deepspeed/checkpoint
      creating build/lib/deepspeed/elasticity
      copying deepspeed/elasticity/__init__.py -> build/lib/deepspeed/elasticity
      copying deepspeed/elasticity/config.py -> build/lib/deepspeed/elasticity
      copying deepspeed/elasticity/constants.py -> build/lib/deepspeed/elasticity
      copying deepspeed/elasticity/elasticity.py -> build/lib/deepspeed/elasticity
      creating build/lib/deepspeed/inference
      copying deepspeed/inference/__init__.py -> build/lib/deepspeed/inference
      copying deepspeed/inference/engine.py -> build/lib/deepspeed/inference
      creating build/lib/deepspeed/launcher
      copying deepspeed/launcher/__init__.py -> build/lib/deepspeed/launcher
      copying deepspeed/launcher/constants.py -> build/lib/deepspeed/launcher
      copying deepspeed/launcher/launch.py -> build/lib/deepspeed/launcher
      copying deepspeed/launcher/multinode_runner.py -> build/lib/deepspeed/launcher
      copying deepspeed/launcher/runner.py -> build/lib/deepspeed/launcher
      creating build/lib/deepspeed/module_inject
      copying deepspeed/module_inject/__init__.py -> build/lib/deepspeed/module_inject
      copying deepspeed/module_inject/inject.py -> build/lib/deepspeed/module_inject
      copying deepspeed/module_inject/module_quantize.py -> build/lib/deepspeed/module_inject
      copying deepspeed/module_inject/replace_module.py -> build/lib/deepspeed/module_inject
      copying deepspeed/module_inject/replace_policy.py -> build/lib/deepspeed/module_inject
      creating build/lib/deepspeed/moe
      copying deepspeed/moe/__init__.py -> build/lib/deepspeed/moe
      copying deepspeed/moe/experts.py -> build/lib/deepspeed/moe
      copying deepspeed/moe/layer.py -> build/lib/deepspeed/moe
      copying deepspeed/moe/sharded_moe.py -> build/lib/deepspeed/moe
      copying deepspeed/moe/utils.py -> build/lib/deepspeed/moe
      creating build/lib/deepspeed/ops
      copying deepspeed/ops/__init__.py -> build/lib/deepspeed/ops
      creating build/lib/deepspeed/pipe
      copying deepspeed/pipe/__init__.py -> build/lib/deepspeed/pipe
      creating build/lib/deepspeed/profiling
      copying deepspeed/profiling/__init__.py -> build/lib/deepspeed/profiling
      copying deepspeed/profiling/config.py -> build/lib/deepspeed/profiling
      copying deepspeed/profiling/constants.py -> build/lib/deepspeed/profiling
      creating build/lib/deepspeed/runtime
      copying deepspeed/runtime/__init__.py -> build/lib/deepspeed/runtime
      copying deepspeed/runtime/bf16_optimizer.py -> build/lib/deepspeed/runtime
      copying deepspeed/runtime/config.py -> build/lib/deepspeed/runtime
      copying deepspeed/runtime/config_utils.py -> build/lib/deepspeed/runtime
      copying deepspeed/runtime/constants.py -> build/lib/deepspeed/runtime
      copying deepspeed/runtime/dataloader.py -> build/lib/deepspeed/runtime
      copying deepspeed/runtime/eigenvalue.py -> build/lib/deepspeed/runtime
      copying deepspeed/runtime/engine.py -> build/lib/deepspeed/runtime
      copying deepspeed/runtime/lr_schedules.py -> build/lib/deepspeed/runtime
      copying deepspeed/runtime/progressive_layer_drop.py -> build/lib/deepspeed/runtime
      copying deepspeed/runtime/quantize.py -> build/lib/deepspeed/runtime
      copying deepspeed/runtime/sparse_tensor.py -> build/lib/deepspeed/runtime
      copying deepspeed/runtime/state_dict_factory.py -> build/lib/deepspeed/runtime
      copying deepspeed/runtime/utils.py -> build/lib/deepspeed/runtime
      copying deepspeed/runtime/weight_quantizer.py -> build/lib/deepspeed/runtime
      creating build/lib/deepspeed/utils
      copying deepspeed/utils/__init__.py -> build/lib/deepspeed/utils
      copying deepspeed/utils/debug.py -> build/lib/deepspeed/utils
      copying deepspeed/utils/distributed.py -> build/lib/deepspeed/utils
      copying deepspeed/utils/exceptions.py -> build/lib/deepspeed/utils
      copying deepspeed/utils/groups.py -> build/lib/deepspeed/utils
      copying deepspeed/utils/logging.py -> build/lib/deepspeed/utils
      copying deepspeed/utils/nvtx.py -> build/lib/deepspeed/utils
      copying deepspeed/utils/timer.py -> build/lib/deepspeed/utils
      copying deepspeed/utils/zero_to_fp32.py -> build/lib/deepspeed/utils
      creating build/lib/deepspeed/autotuning/tuner
      copying deepspeed/autotuning/tuner/__init__.py -> build/lib/deepspeed/autotuning/tuner
      copying deepspeed/autotuning/tuner/base_tuner.py -> build/lib/deepspeed/autotuning/tuner
      copying deepspeed/autotuning/tuner/cost_model.py -> build/lib/deepspeed/autotuning/tuner
      copying deepspeed/autotuning/tuner/index_based_tuner.py -> build/lib/deepspeed/autotuning/tuner
      copying deepspeed/autotuning/tuner/model_based_tuner.py -> build/lib/deepspeed/autotuning/tuner
      copying deepspeed/autotuning/tuner/utils.py -> build/lib/deepspeed/autotuning/tuner
      creating build/lib/deepspeed/ops/adagrad
      copying deepspeed/ops/adagrad/__init__.py -> build/lib/deepspeed/ops/adagrad
      copying deepspeed/ops/adagrad/cpu_adagrad.py -> build/lib/deepspeed/ops/adagrad
      creating build/lib/deepspeed/ops/adam
      copying deepspeed/ops/adam/__init__.py -> build/lib/deepspeed/ops/adam
      copying deepspeed/ops/adam/cpu_adam.py -> build/lib/deepspeed/ops/adam
      copying deepspeed/ops/adam/fused_adam.py -> build/lib/deepspeed/ops/adam
      copying deepspeed/ops/adam/multi_tensor_apply.py -> build/lib/deepspeed/ops/adam
      creating build/lib/deepspeed/ops/aio
      copying deepspeed/ops/aio/__init__.py -> build/lib/deepspeed/ops/aio
      creating build/lib/deepspeed/ops/lamb
      copying deepspeed/ops/lamb/__init__.py -> build/lib/deepspeed/ops/lamb
      copying deepspeed/ops/lamb/fused_lamb.py -> build/lib/deepspeed/ops/lamb
      creating build/lib/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/__init__.py -> build/lib/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/async_io.py -> build/lib/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/builder.py -> build/lib/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/cpu_adagrad.py -> build/lib/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/cpu_adam.py -> build/lib/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/fused_adam.py -> build/lib/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/fused_lamb.py -> build/lib/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/quantizer.py -> build/lib/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/sparse_attn.py -> build/lib/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/stochastic_transformer.py -> build/lib/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/transformer.py -> build/lib/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/transformer_inference.py -> build/lib/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/utils.py -> build/lib/deepspeed/ops/op_builder
      creating build/lib/deepspeed/ops/quantizer
      copying deepspeed/ops/quantizer/__init__.py -> build/lib/deepspeed/ops/quantizer
      copying deepspeed/ops/quantizer/quantizer.py -> build/lib/deepspeed/ops/quantizer
      creating build/lib/deepspeed/ops/sparse_attention
      copying deepspeed/ops/sparse_attention/__init__.py -> build/lib/deepspeed/ops/sparse_attention
      copying deepspeed/ops/sparse_attention/bert_sparse_self_attention.py -> build/lib/deepspeed/ops/sparse_attention
      copying deepspeed/ops/sparse_attention/matmul.py -> build/lib/deepspeed/ops/sparse_attention
      copying deepspeed/ops/sparse_attention/softmax.py -> build/lib/deepspeed/ops/sparse_attention
      copying deepspeed/ops/sparse_attention/sparse_attention_utils.py -> build/lib/deepspeed/ops/sparse_attention
      copying deepspeed/ops/sparse_attention/sparse_self_attention.py -> build/lib/deepspeed/ops/sparse_attention
      copying deepspeed/ops/sparse_attention/sparsity_config.py -> build/lib/deepspeed/ops/sparse_attention
      creating build/lib/deepspeed/ops/transformer
      copying deepspeed/ops/transformer/__init__.py -> build/lib/deepspeed/ops/transformer
      copying deepspeed/ops/transformer/transformer.py -> build/lib/deepspeed/ops/transformer
      creating build/lib/deepspeed/ops/sparse_attention/trsrc
      copying deepspeed/ops/sparse_attention/trsrc/__init__.py -> build/lib/deepspeed/ops/sparse_attention/trsrc
      creating build/lib/deepspeed/ops/transformer/inference
      copying deepspeed/ops/transformer/inference/__init__.py -> build/lib/deepspeed/ops/transformer/inference
      copying deepspeed/ops/transformer/inference/moe_inference.py -> build/lib/deepspeed/ops/transformer/inference
      copying deepspeed/ops/transformer/inference/transformer_inference.py -> build/lib/deepspeed/ops/transformer/inference
      creating build/lib/deepspeed/profiling/flops_profiler
      copying deepspeed/profiling/flops_profiler/__init__.py -> build/lib/deepspeed/profiling/flops_profiler
      copying deepspeed/profiling/flops_profiler/profiler.py -> build/lib/deepspeed/profiling/flops_profiler
      creating build/lib/deepspeed/runtime/activation_checkpointing
      copying deepspeed/runtime/activation_checkpointing/__init__.py -> build/lib/deepspeed/runtime/activation_checkpointing
      copying deepspeed/runtime/activation_checkpointing/checkpointing.py -> build/lib/deepspeed/runtime/activation_checkpointing
      copying deepspeed/runtime/activation_checkpointing/config.py -> build/lib/deepspeed/runtime/activation_checkpointing
      creating build/lib/deepspeed/runtime/comm
      copying deepspeed/runtime/comm/__init__.py -> build/lib/deepspeed/runtime/comm
      copying deepspeed/runtime/comm/coalesced_collectives.py -> build/lib/deepspeed/runtime/comm
      copying deepspeed/runtime/comm/mpi.py -> build/lib/deepspeed/runtime/comm
      copying deepspeed/runtime/comm/nccl.py -> build/lib/deepspeed/runtime/comm
      creating build/lib/deepspeed/runtime/compression
      copying deepspeed/runtime/compression/__init__.py -> build/lib/deepspeed/runtime/compression
      copying deepspeed/runtime/compression/cupy.py -> build/lib/deepspeed/runtime/compression
      creating build/lib/deepspeed/runtime/data_pipeline
      copying deepspeed/runtime/data_pipeline/__init__.py -> build/lib/deepspeed/runtime/data_pipeline
      copying deepspeed/runtime/data_pipeline/curriculum_scheduler.py -> build/lib/deepspeed/runtime/data_pipeline
      creating build/lib/deepspeed/runtime/fp16
      copying deepspeed/runtime/fp16/__init__.py -> build/lib/deepspeed/runtime/fp16
      copying deepspeed/runtime/fp16/fused_optimizer.py -> build/lib/deepspeed/runtime/fp16
      copying deepspeed/runtime/fp16/loss_scaler.py -> build/lib/deepspeed/runtime/fp16
      copying deepspeed/runtime/fp16/unfused_optimizer.py -> build/lib/deepspeed/runtime/fp16
      creating build/lib/deepspeed/runtime/pipe
      copying deepspeed/runtime/pipe/__init__.py -> build/lib/deepspeed/runtime/pipe
      copying deepspeed/runtime/pipe/engine.py -> build/lib/deepspeed/runtime/pipe
      copying deepspeed/runtime/pipe/module.py -> build/lib/deepspeed/runtime/pipe
      copying deepspeed/runtime/pipe/p2p.py -> build/lib/deepspeed/runtime/pipe
      copying deepspeed/runtime/pipe/schedule.py -> build/lib/deepspeed/runtime/pipe
      copying deepspeed/runtime/pipe/topology.py -> build/lib/deepspeed/runtime/pipe
      creating build/lib/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/__init__.py -> build/lib/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/aio_config.py -> build/lib/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/async_swapper.py -> build/lib/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/constants.py -> build/lib/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/optimizer_utils.py -> build/lib/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/partitioned_optimizer_swapper.py -> build/lib/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/partitioned_param_swapper.py -> build/lib/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/pipelined_optimizer_swapper.py -> build/lib/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/utils.py -> build/lib/deepspeed/runtime/swap_tensor
      creating build/lib/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/__init__.py -> build/lib/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/config.py -> build/lib/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/constants.py -> build/lib/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/contiguous_memory_allocator.py -> build/lib/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/linear.py -> build/lib/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/offload_config.py -> build/lib/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/offload_constants.py -> build/lib/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/partition_parameters.py -> build/lib/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/partitioned_param_coordinator.py -> build/lib/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/stage3.py -> build/lib/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/stage_1_and_2.py -> build/lib/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/test.py -> build/lib/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/tiling.py -> build/lib/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/utils.py -> build/lib/deepspeed/runtime/zero
      creating build/lib/deepspeed/runtime/fp16/onebit
      copying deepspeed/runtime/fp16/onebit/__init__.py -> build/lib/deepspeed/runtime/fp16/onebit
      copying deepspeed/runtime/fp16/onebit/adam.py -> build/lib/deepspeed/runtime/fp16/onebit
      copying deepspeed/runtime/fp16/onebit/lamb.py -> build/lib/deepspeed/runtime/fp16/onebit
      copying deepspeed/runtime/fp16/onebit/zoadam.py -> build/lib/deepspeed/runtime/fp16/onebit
      running egg_info
      writing deepspeed.egg-info/PKG-INFO
      writing dependency_links to deepspeed.egg-info/dependency_links.txt
      writing entry points to deepspeed.egg-info/entry_points.txt
      writing requirements to deepspeed.egg-info/requires.txt
      writing top-level names to deepspeed.egg-info/top_level.txt
      reading manifest template 'MANIFEST.in'
      warning: no files found matching '*.hip' under directory 'deepspeed'
      warning: no files found matching '*.cc' under directory 'deepspeed'
      warning: no files found matching '*.tr' under directory 'csrc'
      warning: no files found matching '*.cc' under directory 'csrc'
      adding license file 'LICENSE'
      writing manifest file 'deepspeed.egg-info/SOURCES.txt'
      copying deepspeed/autotuning/.gitignore -> build/lib/deepspeed/autotuning
      copying deepspeed/autotuning/README.md -> build/lib/deepspeed/autotuning
      creating build/lib/deepspeed/autotuning/config_templates
      copying deepspeed/autotuning/config_templates/template_zero0.json -> build/lib/deepspeed/autotuning/config_templates
      copying deepspeed/autotuning/config_templates/template_zero1.json -> build/lib/deepspeed/autotuning/config_templates
      copying deepspeed/autotuning/config_templates/template_zero2.json -> build/lib/deepspeed/autotuning/config_templates
      copying deepspeed/autotuning/config_templates/template_zero3.json -> build/lib/deepspeed/autotuning/config_templates
      error: can't copy 'deepspeed/ops/csrc': doesn't exist or not a regular file
      [end of output]

gabrielhuang avatar Apr 25 '22 18:04 gabrielhuang

I'm not able to reproduce this error on my side. Could you please try copying the csrc directory to where the symlink is? From the top level of the DeepSpeed repo: rm ./deepspeed/ops/csrc && cp -R ./csrc ./deepspeed/ops/

Does this allow you to install with pip install .?

mrwyattii avatar Apr 27 '22 19:04 mrwyattii

I also meet the same error in Linux OS

hanquansanren avatar May 01 '22 06:05 hanquansanren

It looks like the cause is https://github.com/pypa/setuptools_scm/issues/457.

Reproduction steps:

  1. docker run -it --gpus all deepspeed/deepspeed:latest_torch111 bash
    • Probably doesn't exactly have to be latest_torch111.
  2. git clone --depth=1 https://github.com/microsoft/deepspeech.git
  3. cd deepspeech
  4. pip install .

Extending @mrwyattii's suggestion, the following is a valid workaround that allows pip install .:

rm deepspeed/ops/{csrc,op_builder}
cp -R csrc op_builder deepspeed/ops/

A strange thing is that when I do the same thing outside the container (hence just steps 2, 3, and 4), pip install . works fine. I currently don't understand why. I'm on RHEL.

jaywonchung avatar Aug 24 '22 02:08 jaywonchung

It looks like the cause is pypa/setuptools_scm#457.

Reproduction steps:

  1. docker run -it --gpus all deepspeed/deepspeed:latest_torch111 bash

    • Probably doesn't exactly have to be latest_torch111.

  2. git clone --depth=1 https://github.com/microsoft/deepspeech.git

  3. cd deepspeech

  4. pip install .

Extending @mrwyattii's suggestion, the following is a valid workaround that allows pip install .:

rm deepspeed/ops/{csrc,op_builder}
cp -R csrc op_builder deepspeed/ops/

A strange thing is that when I do the same thing outside the container (hence just steps 2, 3, and 4), pip install . works fine. I currently don't understand why. I'm on RHEL.

Thanks, this solved my problem.

SylarTiaNII avatar Jan 04 '23 12:01 SylarTiaNII

Closing this issue since the issue is stale and based on a very old version of DeepSpeed. If you are still hitting this with newer versions of DeepSpeed and pip, please open a new issue and link this one and I'd be happy to take a look.

loadams avatar Aug 14 '23 20:08 loadams