opencompass

opencompass copied to clipboard

[Feature] How to download the entire leaderboard?

5

### Describe the feature Including all the evaluation results shown on the webpage? ### Will you implement it? - [ ] I would like to implement this feature and create...

zhimin-z

[Feature] 多模态榜单

4

### Describe the feature 之前多模态榜单上好像有好几个bench一起评测(类似LLM榜单上好几个数据集综合评测)，现在只有MMBench了。请问其他数据集评测是迁移到哪里了吗？有没有可能恢复回来呀？ ### Will you implement it? - [ ] I would like to implement this feature and create a PR!

lj163ucas

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand...

KevinNuNu

将llama-2-7b-chat 模型转换成 turbomind的形式，应该用哪个配置脚本去测试精度 python3 -m lmdeploy.serve.turbomind.deploy llama2 /models/llama-2-7b-chat

3

### Prerequisite - [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help. - [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass). ### Type...

seeyourcell

### Describe the feature BackLog, add user document for concurrent api usage ### Will you implement it? - [X] I would like to implement this feature and create a PR!

yingfhu

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand...

jingmingzhuo

[Bug] ceval, cmmlu, mmlu 的 gen 对话模板行为不一致，mmlu 的对话模板存在问题

11

### 先决条件 - [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。 - [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。 ### 问题类型 我正在使用官方支持的任务/模型/数据集进行评估。 ### 环境 {'CUDA available': True, 'CUDA_HOME': '/usr/local/cuda', 'GCC': 'gcc (Debian 10.2.1-6) 10.2.1 20210110',...

LiuLinyun

[Feature] 代码相关数据集评测保留更详细的评测信息

2

### 描述该功能 对于HumanEval这样的代码相关的数据集，评测时保存哪些错了哪些对了，以及具体的执行报错信息（traceback或者至少error message，能指向具体的代码行最好），这样方便查找问题，以及排除后处理等的影响、区分一些trivial的错误（如缺少import）等。如果能针对错误类型再做一个类似报表的统计，那就更好了。 ### 是否希望自己实现该功能？ - [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

II-Matto

[Bug] [doc] page not found, parent page: `https://opencompass.readthedocs.io/en/stable/`

1

### 先决条件 - [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。 - [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。 ### 问题类型 我正在使用官方支持的任务/模型/数据集进行评估。 ### 环境 - ### 重现问题 - 代码/配置示例 - ### 重现问题 - 命令或脚本 -...

del-zhenwu

[Bug] Running demo failed with `"LayerNormKernelImpl" not implemented for 'Half'`

1

### Prerequisite - [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help. - [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass). ### Type...

del-zhenwu