LightCompress
LightCompress copied to clipboard
ModelTC
[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.
my calib dataset is c4 It seems that different samples have different lengths and cannot be directly torch.cat(samples, dim=0)。 llmc/llmc/data/dataset/base_dataset.py", line 89, in get_calib_dataset batch = torch.cat(samples, dim=0) RuntimeError: Sizes...
config file: ``` base: seed: &seed 42 model: type: Mixtral path: /models/Mixtral-8x7B-Instruct-v0.1 torch_dtype: auto calib: name: pileval download: False path: /app/llmc/tools/data/calib/wikitext2 n_samples: 128 bs: -1 seq_len: 512 preproc: pileval_awq seed:...
Hello, I'm tring to build with tensorrt, following is the config file: ``` base: seed: &seed 42 model: type: Llama path: /models/Meta-Llama-3-8B-Instruct torch_dtype: auto calib: name: pileval download: False path:...
我用awq_w4a16.yml量化后使用save_lightllm保存模型，然后利用lightllm对保存的模型部署API，使用如下命令 `python -m lightllm.server.api_server --model_dir /path/llama-7B \ --host 0.0.0.0 \ --port 8080 \ --tp 1 \ --max_total_token_num 120000` 可以正常加载，但是使用如下命令进行调用测试 `curl http://127.0.0.1:8080/generate \ -X POST \ -d '{"inputs":"What is AI?","parameters":{"max_new_tokens":17, "frequency_penalty":1}}'...
awq config ``` base: seed: &seed 42 model: type: Qwen2 path: /models/Qwen2-7B-Instruct tokenizer_mode: slow torch_dtype: auto calib: name: pileval download: False path: /app/src/llmc/tools/data/calib/pileval n_samples: 128 bs: -1 seq_len: 512 preproc:...
Dear LLMC team, When trying to run LLMC with `opencompass` evaluation (as explained in your documentation and illustrated in a few example YAML files), exceptions are encountered. I see that...
Hello, Why is the Spinquant branch not merged? Do you plan to merge it anytime soon? Thanks
When using static quantization for activation, we encountered an issue where the mixed-precision quantization fails due to the order of registering activation parameters. This incompatibility disrupts the expected behavior of...
