opencompass

opencompass copied to clipboard

[Bug] Different result on mmlu between opencompass and lm-evaluation-harness

2

### Prerequisite - [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help. - [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass). ### Type...

kirliavc

请问我想对cmmlu这类选择题的模板进行更改，但是system变量无法赋值

1

### Prerequisite - [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help. - [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass). ### Type...

Sweetclover

### Describe the feature allow huggingface.py mode to be passed in command cli. https://github.com/open-compass/opencompass/blob/9db56526382944560dde85b2b026dae82df6a2f7/opencompass/models/huggingface.py#L69 ### Will you implement it? - [ ] I would like to implement this feature and...

nlpcat

### Describe the feature https://github.com/open-compass/opencompass/pull/277 Refactors are also in need, e.g. _openai_function ### Will you implement it? - [ ] I would like to implement this feature and create a...

Leymore

### Prerequisite - [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help. - [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass). ### Type...

cdpath

### 描述该功能 Hello everyone! In order to better cater to the needs of our community, we plan to incorporate a range of diverse NLP datasets into the platform. These datasets...

liushz

Add support for Azure, Palm, Anthropic, Cohere, Hugging Face Llama2 70b Models - using litellm

3

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand...

ishaan-jaff

### 先决条件 - [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。 - [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。 ### 问题类型 我正在使用官方支持的任务/模型/数据集进行评估。 ### 环境 参考opencompass docs配置 ### 重现问题 - 代码/配置示例 同官方 ### 重现问题 - 命令或脚本...

Li-jiaxian

[Bug] 评测 Hellaswag 失败

2

### Prerequisite - [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help. - [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass). ### Type...

SefaZeng

[Bug] 根据官方的Quick Start运行，输出结果为空

17

### Prerequisite - [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help. - [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass). ### Type...

MansonHua