LightCompress icon indicating copy to clipboard operation
LightCompress copied to clipboard
Published 1 month ago • verified publisher icon ModelTC

Error while enabling transformed eval_pos

Open dittops opened this issue 9 months ago • 0 comments

I'm trying to test this with RTN. But while named eval for transformed I'm getting following datatype error.

This is my config file

    seed: &seed 42
model:
    type: Llama
    path: meta-llama/Llama-3.2-1B-Instruct
    torch_dtype: auto
eval:
    eval_pos: [pretrain, fake_quant, transformed]
    name: wikitext2
    download: True
    path: eval data path
    seq_len: 2048
    # For 7B / 13B model eval, bs can be set to "1", and inference_per_block can be set to "False".
    # For 70B model eval, bs can be set to "20", and inference_per_block can be set to "True".
    bs: 1
    inference_per_block: False
quant:
    method: RTN
    weight:
        bit: 8
        symmetric: True
        granularity: per_channel
        group_size: -1
    act:
        bit: 8
        symmetric: True
        granularity: per_token
save:
    save_fake: False
    save_path: save/
    save_vllm: True

This is error I'm getting

2025-03-13 09:51:33.647 | INFO     | llmc.eval.eval_ppl:eval_func:23 - index : 1/141
[rank0]: Traceback (most recent call last):
[rank0]:   File "<frozen runpy>", line 198, in _run_module_as_main
[rank0]:   File "<frozen runpy>", line 88, in _run_code
[rank0]:   File "/home/sdp/ditto/llmc/llmc/__main__.py", line 274, in <module>
[rank0]:     main(config, args.use_kubernetes)
[rank0]:   File "/home/sdp/ditto/llmc/llmc/__main__.py", line 171, in main
[rank0]:     quant_eval_result = eval_model(model, blockwise_opts, eval_list, eval_pos='transformed')
[rank0]:                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/llmc/eval/utils.py", line 88, in eval_model
[rank0]:     res = eval_class.eval(model)
[rank0]:           ^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/llmc/eval/eval_base.py", line 208, in eval
[rank0]:     eval_res = self.eval_func(
[rank0]:                ^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/llmc/eval/eval_ppl.py", line 32, in eval_func
[rank0]:     lm_logits = model.model(inputs).logits
[rank0]:                 ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 842, in forward
[rank0]:     outputs = self.model(
[rank0]:               ^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 594, in forward
[rank0]:     layer_outputs = decoder_layer(
[rank0]:                     ^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 336, in forward
[rank0]:     hidden_states, self_attn_weights = self.self_attn(
[rank0]:                                        ^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 270, in forward
[rank0]:     query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/.env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/sdp/ditto/llmc/llmc/compression/quantization/module_utils.py", line 318, in forward
[rank0]:     y = torch.functional.F.linear(x, self.weight, self.bias)
[rank0]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: expected m1 and m2 to have the same dtype, but got: c10::BFloat16 != signed char
E0313 09:51:34.483000 85271 .env/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 85349) of binary: /home/sdp/ditto/llmc/.env/bin/python3.11```

dittops avatar Mar 13 '25 10:03 dittops