Bug: Your fork of opencompass is inconsistent with latest LLMC modifications

[sasha-hailo]

Dear LLMC team, When trying to run LLMC with opencompass evaluation (as explained in your documentation and illustrated in a few example YAML files), exceptions are encountered. I see that the way models and block-optimization algorithm objects are instantiated today underwent changes in LLMC repo. These changes are not reflected in your version of opencompass.

The existing code in file huggingface_above_v4_33.py is

                model = MODEL_REGISTRY[kwargs['model']["type"]](path, kwargs['model']["torch_dtype"], device_map="auto", use_cache=True)
                blockwise_opt = ALGO_REGISTRY[kwargs["quant"]["method"]](
                    model, quant_config=kwargs["quant"], input=None, config=None
                )

While the new LLMC code is

model = MODEL_REGISTRY[config.model.type](config)

blockwise_opt = ALGO_REGISTRY[modality_config.method](
	model,
	modality_config,
	input=None,
	padding_mask=None,
	config=config,
)

I suspect there are additional issues I'm unaware of, but in short, opencompass does not work now. Can you please look into it?

Thanks in advance!

[ztisbest]

try:
    from llmc.compression.quantization import *
    from llmc.models import *
    from llmc.utils.registry_factory import ALGO_REGISTRY, MODEL_REGISTRY
    from llmc.utils.utils import get_modality

            if kwargs['is_quant']:
                print("is_quant is True")
                from mmengine import Config
                config = Config(kwargs)
                _, modality_configs = get_modality(config)
                model = MODEL_REGISTRY[kwargs['model']['type']](config, device_map="auto", use_cache=True)
                blockwise_opt = ALGO_REGISTRY[kwargs["quant"]['method']](
                    model, modality_configs[0], input=None, padding_mask=None, config=config
                )

you can replace the following code in [opencampass/opencampass/models/huggingface_above_v4_33.py] with this code

            if kwargs['is_quant']:
                print("is_quant is True")
                model = MODEL_REGISTRY[kwargs['model']["type"]](path, kwargs['model']["torch_dtype"], device_map="auto", use_cache=True)
                blockwise_opt = ALGO_REGISTRY[kwargs["quant"]["method"]](
                    model, quant_config=kwargs["quant"], input=None, config=None
                )

I tested it with the opt-125m model, and it was effective. And I find [llmc/models/opt.py] need add the following code, if you use other models,maybe also need this.

class Opt(BaseModel):
    def __init__(self, config, device_map=None, use_cache=False):
        super().__init__(config, device_map, use_cache)
        self.find_blocks()
        self.find_embed_layers()
        self.find_block_name()