AttributeError: 'BloomForCausalLM' object has no attribute 'module'

[detectiveJoshua]

I was test the deepspeed bloom inference with bloom-7b1 using 3 GPUs. This error didn't occur when I ran the accelerate inference. command line:

deepspeed --include localhost:2,5,6 --module inference_server.benchmark --model_name ./bloom-7b1 --model_class AutoModelForCausalLM --dtype fp32 --deployment_framework ds_inference --benchmark_cycles 5

log:

Traceback (most recent call last): 
  File "/home/qiuziheng/anaconda3/envs/ml/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/qiuziheng/anaconda3/envs/ml/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/benchmark.py", line 119, in <module>
    main()
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/benchmark.py", line 115, in main
    benchmark_end_to_end(args)
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/benchmark.py", line 48, in benchmark_end_to_end
    model, initialization_time = run_and_log_time(partial(ModelDeployment, args=args, grpc_allowed=False))
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/utils/utils.py", line 152, in run_and_log_time
    results = execs()
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/model_handler/deployment.py", line 54, in __init__
Traceback (most recent call last):
  File "/home/qiuziheng/anaconda3/envs/ml/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/qiuziheng/anaconda3/envs/ml/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/benchmark.py", line 119, in <module>
    main()
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/benchmark.py", line 115, in main
    benchmark_end_to_end(args)
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/benchmark.py", line 48, in benchmark_end_to_end
    model, initialization_time = run_and_log_time(partial(ModelDeployment, args=args, grpc_allowed=False))
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/utils/utils.py", line 152, in run_and_log_time
    results = execs()
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/model_handler/deployment.py", line 54, in __init__
    self.model = get_model_class(args.deployment_framework)(args)
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/models/ds_inference.py", line 65, in __init__
    self.model = get_model_class(args.deployment_framework)(args)
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/models/ds_inference.py", line 65, in __init__
Traceback (most recent call last):
  File "/home/qiuziheng/anaconda3/envs/ml/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/qiuziheng/anaconda3/envs/ml/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/benchmark.py", line 119, in <module>
    main()
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/benchmark.py", line 115, in main
    benchmark_end_to_end(args)
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/benchmark.py", line 48, in benchmark_end_to_end
    self.model = self.model.module
  File "/home/qiuziheng/anaconda3/envs/ml/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1269, in __getattr__
    model, initialization_time = run_and_log_time(partial(ModelDeployment, args=args, grpc_allowed=False))
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/utils/utils.py", line 152, in run_and_log_time
    results = execs()
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/model_handler/deployment.py", line 54, in __init__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'BloomForCausalLM' object has no attribute 'module'
    self.model = self.model.module
  File "/home/qiuziheng/anaconda3/envs/ml/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1269, in __getattr__
    self.model = get_model_class(args.deployment_framework)(args)
  File "/nasdata/qiuziheng/transformers-bloom-inference/inference_server/models/ds_inference.py", line 65, in __init__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'BloomForCausalLM' object has no attribute 'module'
    self.model = self.model.module
  File "/home/qiuziheng/anaconda3/envs/ml/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1269, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'BloomForCausalLM' object has no attribute 'module'
[2023-06-01 14:42:29,428] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 314922
[2023-06-01 14:42:29,429] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 314925
[2023-06-01 14:42:29,468] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 314926
[2023-06-01 14:42:29,498] [ERROR] [launch.py:434:sigkill_handler] ['/home/qiuziheng/anaconda3/envs/ml/bin/python', '-u', '-m', 'inference_server.benchmark', '--local_rank=2', '--model_name', './bloom-7b1', '--model_class', 'AutoModelForCausalLM', '--dtype', 'fp32', '--deployment_framework', 'ds_inference', '--benchmark_cycles', '5'] exits with return code = 1

Environment:

pytorch==1.13.1
transformers==4.29.2
deepspeed==0.9.2