opencompass

opencompass copied to clipboard

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand...

bittersweet1999

### 先决条件 - [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。 - [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。 ### 问题类型 我正在使用官方支持的任务/模型/数据集进行评估。 ### 环境 {'CUDA available': True, 'CUDA_HOME': None, 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0', 'GPU...

hailsham

### Prerequisite - [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help. - [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass). ### Type...

watermelon-lee

[Bug] ds1000数据集eval阶段报错

1

### 先决条件 - [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。 - [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。 ### 问题类型 我正在使用官方支持的任务/模型/数据集进行评估。 ### 环境 {'CUDA available': True, 'CUDA_HOME': '/usr/local/cuda', 'GCC': 'gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0', 'GPU...

bayma-1

### 描述该功能 我已经在自己的脚本里有一个量化后的model实例了，我如何启动直接启动评测而不是重新在命令行中用python run.py方法加载model 就跟lm-eval支持的一样  ### 是否希望自己实现该功能？ - [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

zitgit

### Discussed in https://github.com/open-compass/opencompass/discussions/1347 Originally posted by **starplatinum3** July 22, 2024 求支持OpenBuddy GitHub - OpenBuddy/OpenBuddy: Open Multilingual Chatbot for Everyone https://github.com/OpenBuddy/OpenBuddy

tonysy

### Describe the feature 目前评测多个数据集时，如果不使用vllm，只能在模型测添加batch_size，但是有的数据集较长，有的较短，同样的batch_size可能会利用gpu不充分，如何针对数据集设置batch_size。 ### Will you implement it? - [ ] I would like to implement this feature and create a PR!

JBoRu

### Describe the feature Improve the inference speech with vLLM batch API ### Will you implement it? - [ ] I would like to implement this feature and create a...

tonysy

## Motivation There is no need to use `torchrun` for single GPU inference. Besides, `python` runner is more friendly for debugging. The debugging snippet is as follows https://github.com/open-compass/opencompass/blob/889e7e11409d83fe312ecc7d7f0ed8861a84cc92/opencompass/runners/local.py#L116-L131

xu-song

### Prerequisite - [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help. - [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass). ### Type...

QiMingChina