DeepSpeed
DeepSpeed copied to clipboard
[BUG] AttributeError: module 'deepspeed.utils' has no attribute 'zero_to_fp32'
There is no zero_to_fp32 in my installation. I tried pip and install from source, but both did not contain zero_to_fp32.
Below is copied from my terminal. I first imported deepspeed and tried to use zero_to_fp32 under utils module. So I typed 'deepspeed.utils.' then I pressed tab key twice and there was no zero_to_fp32.
Screenshots
Python 3.7.13 (default, Mar 29 2022, 02:18:16)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import deepspeed
>>> deepspeed.utils.
deepspeed.utils.DeprecatedException( deepspeed.utils.expert_tensor_parallel_world_size deepspeed.utils.initialize( deepspeed.utils.mpu
deepspeed.utils.OnDevice( deepspeed.utils.fragment_address( deepspeed.utils.instrument_w_nvtx( deepspeed.utils.nvtx
deepspeed.utils.RepeatingLoader( deepspeed.utils.get_caller_func( deepspeed.utils.link_hp_params( deepspeed.utils.tensor_fragment(
deepspeed.utils.comms_logging deepspeed.utils.get_full_hp_param( deepspeed.utils.log_dist( deepspeed.utils.timer
deepspeed.utils.debug deepspeed.utils.get_hp_fragment_mapping( deepspeed.utils.logger deepspeed.utils.types
deepspeed.utils.dist deepspeed.utils.groups deepspeed.utils.logging
deepspeed.utils.exceptions deepspeed.utils.init_on_device deepspeed.utils.mixed_precision_linkage
System info (please complete the following information):
- OS: Ubuntu 16.04
- GPU count and types: 1080ti x 4
- Python version 3.7
Hi @root20, I could not reproduce this.
What is your ds_report? Mine is
deepspeed info ................... 0.8.1+4af1f76a, 4af1f76a, master
torch cuda version ............... 11.6
torch hip version ................ None
nvcc version ..................... 11.6
deepspeed wheel compiled w. ...... torch 1.12, cuda 11.6
This is mine.
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] async_io: please install the libaio-dev package with apt
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
[WARNING] please install triton==1.0.0 if you want to use sparse attention
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/opt/conda/lib/python3.7/site-packages/torch']
torch version .................... 1.13.1+cu117
deepspeed install path ........... ['/opt/conda/lib/python3.7/site-packages/deepspeed']
deepspeed info ................... 0.8.0, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.4
deepspeed wheel compiled w. ...... torch 1.13, cuda 11.7
Could you try import deepspeed.utils.zero_to_fp32
to see if zero_to_fp32 can be used in that way.
It works as expected. thank you!