[BUG] undefined symbol: _ZNK3c107SymBool10guard_boolEPKcl

[anonNo2]

Describe the bug

Traceback (most recent call last):
  File "deepspeed_inference_demo.py", line 16, in <module>
    ds_model = deepspeed.init_inference(
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/__init__.py", line 333, in init_inference
    engine = InferenceEngine(model, config=ds_inference_config)
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 194, in __init__
    self._apply_injection_policy(config)
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 396, in _apply_injection_policy
    replace_transformer_layer(client_module, self.module, checkpoint, config, self.config)
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 498, in replace_transformer_layer
    replaced_module = replace_module(model=model,
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 731, in replace_module
    replaced_module, _ = _replace_module(model, policy)
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 756, in _replace_module
    _, layer_id = _replace_module(child, policies, layer_id=layer_id)
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 756, in _replace_module
    _, layer_id = _replace_module(child, policies, layer_id=layer_id)
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 748, in _replace_module
    replaced_module = policies[child.__class__][0](child, policies[child.__class__][-1], layer_id)
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 488, in replace_fn
    new_module = replace_with_policy(child,
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 346, in replace_with_policy
    _container.create_module()
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/module_inject/containers/bloom.py", line 27, in create_module
    self.module = DeepSpeedBloomInference(_config, mp_group=self.mp_group)
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/model_implementations/transformers/ds_bloom.py", line 20, in __init__
    super().__init__(config, mp_group, quantize_scales, quantize_groups, merge_count, mlp_extra_grouping)
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/model_implementations/transformers/ds_transformer.py", line 54, in __init__
    inference_cuda_module = builder.load()
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 443, in load
    return importlib.import_module(self.absolute_name())
  File "/root/anaconda3/envs/py38deepspeed/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 657, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1166, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed/ops/transformer/inference/transformer_inference_op.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZNK3c107SymBool10guard_boolEPKcl

To Reproduce Steps to reproduce the behavior:

# Load Model and Tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name,padding_side="right", use_fast=True,)
# we use device_map auto to automatically place all shards on the GPU to save CPU memory
model = AutoModelForCausalLM.from_pretrained(model_name, low_cpu_mem_usage=True, torch_dtype=torch.float16).cuda()




# init deepspeed inference engine
ds_model = deepspeed.init_inference(
    model=model,      # Transformers models
    mp_size=1,        # Number of GPU
    dtype=torch.float16, # dtype of the weights (fp16)
    replace_method="auto", # Lets DS autmatically identify the layer to replace
    replace_with_kernel_inject=True, # replace the model with the kernel injector
)
print(f"model is loaded on device {ds_model.module.device}")

ds_report output

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
async_io ............... [YES] ...... [OKAY]
cpu_adagrad ............ [YES] ...... [OKAY]
cpu_adam ............... [YES] ...... [OKAY]
fused_adam ............. [YES] ...... [OKAY]
fused_lamb ............. [YES] ...... [OKAY]
quantizer .............. [YES] ...... [OKAY]
random_ltd ............. [YES] ...... [OKAY]
sparse_attn ............ [YES] ...... [OKAY]
spatial_inference ...... [YES] ...... [OKAY]
transformer ............ [YES] ...... [OKAY]
stochastic_transformer . [YES] ...... [OKAY]
transformer_inference .. [YES] ...... [OKAY]
utils .................. [YES] ...... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/torch']
torch version .................... 1.13.1+cu117
deepspeed install path ........... ['/root/anaconda3/envs/py38deepspeed/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.9.2+2e99f6ed, 2e99f6ed, master
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.7
deepspeed wheel compiled w. ...... torch 1.13, cuda 11.7

Screenshots If applicable, add screenshots to help explain your problem.

System info (please complete the following information):

• OS: [e.g. Ubuntu 22.04]

• GPU count and types [e.g. single machine with x8 A40s]

• Python version 3.8

[DurbinLiu]

I have met a similar bug, like 'cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZNK3c107SymBool10guard_boolEPKcl' in another project.I solved it by recompiling C++/CUDA extension, using command like python setup.py develop. Hope this will be helpful for u.

[libaolong4473]

I have met a similar bug, like 'cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZNK3c107SymBool10guard_boolEPKcl' in another project.I solved it by recompiling C++/CUDA extension, using command like python setup.py develop. Hope this will be helpful for u.

Can you provide a detailed description of your process of solving this problem, instead of using the term like