[BUG] AttributeError: module 'deepspeed.utils' has no attribute 'zero_to_fp32'

[root20]

There is no zero_to_fp32 in my installation. I tried pip and install from source, but both did not contain zero_to_fp32.

Below is copied from my terminal. I first imported deepspeed and tried to use zero_to_fp32 under utils module. So I typed 'deepspeed.utils.' then I pressed tab key twice and there was no zero_to_fp32.

Screenshots

Python 3.7.13 (default, Mar 29 2022, 02:18:16) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import deepspeed
>>> deepspeed.utils.
deepspeed.utils.DeprecatedException(               deepspeed.utils.expert_tensor_parallel_world_size  deepspeed.utils.initialize(                        deepspeed.utils.mpu
deepspeed.utils.OnDevice(                          deepspeed.utils.fragment_address(                  deepspeed.utils.instrument_w_nvtx(                 deepspeed.utils.nvtx
deepspeed.utils.RepeatingLoader(                   deepspeed.utils.get_caller_func(                   deepspeed.utils.link_hp_params(                    deepspeed.utils.tensor_fragment(
deepspeed.utils.comms_logging                      deepspeed.utils.get_full_hp_param(                 deepspeed.utils.log_dist(                          deepspeed.utils.timer
deepspeed.utils.debug                              deepspeed.utils.get_hp_fragment_mapping(           deepspeed.utils.logger                             deepspeed.utils.types
deepspeed.utils.dist                               deepspeed.utils.groups                             deepspeed.utils.logging                            
deepspeed.utils.exceptions                         deepspeed.utils.init_on_device                     deepspeed.utils.mixed_precision_linkage        

System info (please complete the following information):

• OS: Ubuntu 16.04
• GPU count and types: 1080ti x 4
• Python version 3.7

[ShijieZZZZ]

Hi @root20, I could not reproduce this.

What is your ds_report? Mine is

deepspeed info ................... 0.8.1+4af1f76a, 4af1f76a, master
torch cuda version ............... 11.6
torch hip version ................ None
nvcc version ..................... 11.6
deepspeed wheel compiled w. ...... torch 1.12, cuda 11.6

[root20]

This is mine.

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  please install triton==1.0.0 if you want to use sparse attention
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/opt/conda/lib/python3.7/site-packages/torch']
torch version .................... 1.13.1+cu117
deepspeed install path ........... ['/opt/conda/lib/python3.7/site-packages/deepspeed']
deepspeed info ................... 0.8.0, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.4
deepspeed wheel compiled w. ...... torch 1.13, cuda 11.7

[ShijieZZZZ]

Could you try import deepspeed.utils.zero_to_fp32 to see if zero_to_fp32 can be used in that way.

[root20]

It works as expected. thank you!